Method and apparatus for decoding/encoding a video signal

ABSTRACT

The method includes the steps of receiving the multiview video data stream including a random access picture including a random access slice, the random access slice referencing only slice corresponding to a same time and a different view of the random access picture, obtaining identification information representing the multi-view video data stream including initialization information of a reference picture list for the random access slice, obtaining initialization information of a reference picture list for the random access slice based on the identification information, the initialization information representing a reference relation between a plurality of views with view number information and view identification information, initializing the reference picture list using the view number information and the view identification information, obtaining a difference value from the a multi-view video data stream according to the type information, the difference value representing a residual of inter-view reference index in the initialized reference picture list, determining an assignment modification value for modifying the inter-view reference index in the initialized reference picture list according to the difference value, modifying the initialized reference picture list for inter-view prediction using the determined assignment modification value, determining a prediction value of a macroblock in the random access picture based on the modified reference picture list, and decoding the macroblock using the prediction value, wherein the initialization information is obtained from an extension area of a sequence header.

PRIORITY INFORMATION

This application is a continuation of U.S. application Ser. No.12/225,682 filed May 20, 2009, which claims priority from PCTInternational Application No. PCT/KR2007/001583 filed on Mar. 30, 2007,which claims priority on U.S. provisional application Nos. 60/787,171;60/801,398; 60/810,642; 60/830,601; 60/832,153; 60/837,925; 60/840,032;and 60/842,152; filed on Mar. 30, 2006, May 19, 2006, Jun. 5, 2006, Jul.14, 2006, Jul. 21, 2006, Aug. 16, 2006, Aug. 25, 2006, and Sep. 5, 2006.The entire contents of ball of these applications are incorporatedherein by reference.

BACKGROUND Related Art

Compression encoding means a series of signal processing techniques fortransmitting digitalized information via communication circuit orstoring digitalized information in a form suitable for a storage medium.Objects for the compression encoding include audio, video, text, and thelike. In particular, a technique for performing compression encoding ona sequence is called video sequence compression. The video sequence isgenerally characterized in having spatial redundancy and temporalredundancy.

SUMMARY

The present invention is related to encoding and/or decoding a videosignal.

In one embodiment, video data of views other than a current view areused to encode video data and/or decode video data.

In one embodiment, the method includes the steps of receiving themultiview video data stream including a random access picture includinga random access slice, the random access slice referencing only slicecorresponding to a same time and a different view of the random accesspicture, obtaining identification information representing themulti-view video data stream including initialization information of areference picture list for the random access slice, obtaininginitialization information of a reference picture list for the randomaccess slice based on the identification information, the initializationinformation representing a reference relation between a plurality ofviews with view number information and view identification information,initializing the reference picture list using the view numberinformation and the view identification information, obtaining adifference value from the a multi-view video data stream according tothe type information, the difference value representing a residual ofinter-view reference index in the initialized reference picture list,determining an assignment modification value for modifying theinter-view reference index in the initialized reference picture listaccording to the difference value, modifying the initialized referencepicture list for inter-view prediction using the determined assignmentmodification value, determining a prediction value of a macroblock inthe random access picture based on the modified reference picture list,and decoding the macroblock using the prediction value, wherein theinitialization information is obtained from an extension area of asequence header.

In another embodiment, the view number information indicates a number ofreference views of the random access picture, and the viewidentification information provides a view identifier of each referenceview for the random access picture.

In another embodiment, the multi-view video data includes video data ofa base view independent of other views, the base view being a viewdecoded without using inter-view prediction.

In another embodiment, the identification information is obtained basedon NAL (Network Abstract Layer) unit type, the NAL unit type indicatingsequence information of the multi-view video data.

In another embodiment, the difference value is obtained from a sliceheader.

In another embodiment, the determined assignment modification value isused to assign an inter-view reference index to the random accesspicture in the initialized reference picture list.

In another embodiment, the assignment modification value represents avariable associated with a view identifier of an inter-view referencepicture in the initialized reference picture list.

In another embodiment, the position of any other remaining pictures areshifted to later in the initialized reference picture list.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram of an apparatus for decoding a videosignal according to an embodiment.

FIG. 2 is a diagram of configuration information for a multi-view videoaddable to a multi-view video coded bit stream according to anembodiment of the present invention.

FIG. 3 is an internal block diagram of a reference picture listconstructing unit 620 according to an embodiment of the presentinvention.

FIG. 4 is a diagram of a hierarchical structure of level information forproviding view scalability of a video signal according to an embodimentof the present invention.

FIG. 5 is a diagram of a NAL-unit configuration including levelinformation within an extension area of a NAL header according to oneembodiment of the present invention.

FIG. 6 is a diagram of an overall predictive structure of a multi-viewvideo signal according to an embodiment of the present invention toexplain a concept of an inter-view picture group.

FIG. 7 is a diagram of a predictive structure according to an embodimentof the present invention to explain a concept of a newly definedinter-view picture group.

FIG. 8 is a schematic block diagram of an apparatus for decoding amulti-view video using inter-view picture group identificationinformation according to an embodiment of the present invention.

FIG. 9 is a flowchart of a process for constructing a reference picturelist according to an embodiment of the present invention.

FIG. 10 is a diagram to explain a method of initializing a referencepicture list when a current slice is a P-slice according to oneembodiment of the present invention.

FIG. 11 is a diagram to explain a method of initializing a referencepicture list when a current slice is a B-slice according to oneembodiment of the present invention.

FIG. 12 is an internal block diagram of the reference picture listreordering unit shown in FIG. 3 according to an embodiment of thepresent invention.

FIG. 13 is an internal block diagram of a reference index assignmentchanging unit shown in FIG. 12 according to one embodiment of thepresent invention.

FIG. 14 is a diagram to explain a process for reordering a referencepicture list using view information according to one embodiment of thepresent invention.

FIG. 15 is an internal block diagram of a reference picture listreordering unit shown in FIG. 3 according to another embodiment of thepresent invention.

FIG. 16 is an internal block diagram of a reference picture listreordering unit shown in FIG. 15 for inter-view prediction according toan embodiment of the present invention.

FIG. 17 and FIG. 18 are diagrams of syntax for reference picture listreordering according to one embodiment of the present invention.

FIG. 19 is a diagram of syntax for reference picture list reorderingaccording to another embodiment of the present invention.

FIG. 20 is a diagram for a process for obtaining an illuminationdifference value of a current block according to one embodiment of thepresent invention.

FIG. 21 is a flowchart of a process for performing illuminationcompensation of a current block according to an embodiment of thepresent invention.

FIG. 22 is a diagram of a process for obtaining an illuminationdifference prediction value of a current block using information for aneighboring block according to one embodiment of the present invention.

FIG. 23 is a flowchart of a process for performing illuminationcompensation using information for a neighboring block according to oneembodiment of the present invention.

FIG. 24 is a flowchart of a process for performing illuminationcompensation using information for a neighboring block according toanother embodiment of the present invention.

FIG. 25 is a diagram of a process for predicting a current picture usinga picture in a virtual view according to one embodiment of the presentinvention.

FIG. 26 is a flowchart of a process for synthesizing a picture in avirtual view in performing an inter-view prediction in MVC according toan embodiment of the present invention.

FIG. 27 is a flowchart of a method of executing a weighted predictionaccording to a slice type in video signal coding according to thepresent invention.

FIG. 28 is a diagram of macroblock types allowable in a slice type invideo signal coding according to the present invention.

FIG. 29 and FIG. 30 are diagrams of syntax for executing a weightedprediction according to a newly defined slice type according to oneembodiment of the present invention.

FIG. 31 is a flowchart of a method of executing a weighted predictionusing flag information indicating whether to execute inter-view weightedprediction in video signal coding according to the present invention.

FIG. 32 is a diagram to explain a weighted prediction method accordingto flag information indicating whether to execute a weighted predictionusing information for a picture in a view different from that of acurrent picture according to one embodiment of the present invention.

FIG. 33 is a diagram of syntax for executing a weighted predictionaccording to a newly defined flag information according to oneembodiment of the present invention.

FIG. 34 is a flowchart of a method of executing a weighted predictionaccording to a NAL (network abstraction layer) unit type according to anembodiment of the present invention.

FIG. 35 and FIG. 36 are diagrams of syntax for executing a weightedprediction in case that a NAL unit type is for multi-view video codingaccording to one embodiment of the present invention.

FIG. 37 is a partial block diagram of a video signal decoding apparatusaccording to a newly defined slice type according to an embodiment ofthe present invention.

FIG. 38 is a flowchart to explain a method of decoding a video signal inthe apparatus shown in FIG. 37 according to the present invention.

FIG. 39 is a diagram of a macroblock prediction mode according to oneembodiment of the present invention.

FIG. 40 and FIG. 41 are diagrams of syntax having slice type andmacroblock mode applied thereto according to the present invention.

FIG. 42 is a diagram of embodiments to which the slice types in FIG. 41are applied.

FIG. 43 is a diagram of various embodiments of the slice type includedin the slice types shown in FIG. 41.

FIG. 44 is a diagram of a macroblock allowable for a mixed slice type byprediction of two mixed predictions according to one embodiment of thepresent invention.

FIGS. 45 to 47 are diagrams of a macroblock type of a macroblockexisting in a mixed slice by prediction of two mixed predictionsaccording to one embodiment of the present invention.

FIG. 48 is a partial block diagram of a video signal encoding apparatusaccording to a newly defined slice type according to an embodiment ofthe present invention.

FIG. 49 is a flowchart of a method of encoding a video signal in theapparatus shown in FIG. 48 according to the present invention.

DETAIL DESCRIPTION OF EXAMPLE EMBODIMENTS

Example embodiments will now be described more fully with reference tothe accompanying drawings. However, example embodiments may be embodiedin many different forms and should not be construed as being limited tothe example embodiments set forth herein. Example embodiments areprovided so that this disclosure will be thorough, and will fully conveythe scope to those who are skilled in the art. In some exampleembodiments, well-known processes, well-known device structures, andwell-known technologies are not described in detail to avoid the unclearinterpretation of the example embodiments. Throughout the specification,like reference numerals in the drawings denote like elements.

The technique of compressing and encoding video signal data considersspatial redundancy, temporal redundancy, scalable redundancy, andinter-view redundancy. And, it is also able to perform a compressioncoding by considering a mutual redundancy between views in thecompression encoding process. The technique for the compression coding,which considers the inter-view redundancy, is just an embodiment of thepresent invention. And, the technical ideas of the embodiments of thepresent invention are applicable to temporal redundancy, scalableredundancy, etc.

Looking into a configuration of a bit stream in H.264/AVC, there existsa separate layer structure called a NAL (network abstraction layer)between a VCL (video coding layer) dealing with a moving pictureencoding process itself and a lower system that transports and storesencoded information. An output from an encoding process is VCL data andis mapped by NAL units prior to transport or storage. Each NAL unitincludes compressed video data or RBSP (raw byte sequence payload:result data of moving picture compression) that is the datacorresponding to header information.

As shown in FIG. 2, the NAL unit basically includes a NAL header and anRBSP. The NAL header includes flag information (nal_ref_idc) indicatingwhether a slice as a reference picture of the NAL unit is included andan identifier (nal_unit_type) indicating a type of the NAL unit.Compressed original data is stored in the RBSP. And, RBSP trailing bitsare added to a last portion of the RBSP to represent a length of theRBSP as an 8-bit multiplication. As the type of the NAL unit, there isIDR (instantaneous decoding refresh) picture, SPS (sequence parameterset), PPS (picture parameter set), SEI (supplemental enhancementinformation), or the like.

Typically, restrictions for various profiles and levels are set toenable implementation of a target product with an appropriate cost. Inthis case, a decoder should meet the restriction decided according thecorresponding profile and level. Thus, two concepts, ‘profile’ and‘level’ are defined to indicate a function or parameter for representinghow far the decoder can cope with a range of a compressed sequence. And,a profile indicator (profile_idc) can identify that a bit stream isbased on a prescribed profile. The profile indicator means a flagindicating a profile on which a bit stream is based. For instance, inH.264/AVC, if a profile indicator is 66, it means that a bit stream isbased on a baseline profile. If a profile indicator is 77, it means thata bit stream is based on a main profile. If a profile indicator is 88,it means that a bit stream is based on an extended profile. And, theprofile identifier can be included in a sequence parameter set.

So, in order to deal with a multi-view video, whether a profile of aninputted bit stream is a multi-view profile is identified. If theprofile of the inputted bit stream is the multi-view profile, it isdesirable to add syntax to enable at least one additional informationfor multi-view to be transmitted. In this case, the multi-view profileindicates a profile mode handling multi-view video as an amendmenttechnique of H.264/AVC. In MVC, it may be more efficient to add syntaxas additional information for an MVC mode rather than as unconditionalsyntax. For instance, when a profile indicator of AVC indicates amulti-view profile, information for a multi-view video may be added toenhance encoding efficiency.

A sequence parameter set indicates header information, which containsinformation covering coding of an overall sequence such as a profile, alevel, and the like. A whole compressed moving picture (i.e., asequence) should begin at a sequence header. So, a sequence parameterset corresponding to header information should arrive at a decoderbefore data referring to the parameter set arrives. Namely, the sequenceparameter set RBSP plays a role as the header information for the resultdata of the moving picture compression. Once a bit stream is inputted, aprofile indicator identifies on which one of a plurality of profilesthat the inputted bit stream is based. So, by adding a part for decidingwhether an inputted bit stream relates to a multi-view profile (e.g.,‘If (profile_idc==MULTI_VIEW_PROFILE)’) to the syntax, it may be decidedwhether the inputted bit stream relates to the multi-view profile.Various kinds of configuration information may be added only if theinputted bit stream is determined as relating to the multi-view profile.For instance, a number of total views, a number of inter-view referencepictures (List0/1) in case of an inter-view picture group, a number ofinter-view reference pictures (List0/1) in case of a non-inter-viewpicture group, and/or the like may be added. And, various informationfor views are usable for generation and management of a referencepicture list in a decoded picture buffer.

FIG. 1 is a schematic block diagram of an apparatus for decoding a videosignal according to an embodiment of the present invention.

Referring to FIG. 1, the apparatus for decoding a video signal includesa NAL parser 100, an entropy decoding unit 200, an inversequantization/inverse transform unit 300, an intra-prediction unit 400, adeblocking filter unit 500, a decoded picture buffer unit 600, aninter-prediction unit 700, and/or the like.

The decoded picture buffer unit 600 includes a reference picture storingunit 610, a reference picture list constructing unit 620, a referencepicture managing unit 650, and/or the like. And, the reference picturelist constructing unit 620 includes, as shown in FIG. 3, a variablederiving unit 625, a reference picture list initializing unit 630,and/or a reference picture list reordering unit 640.

Returning the FIG. 1, the inter-prediction unit 700 includes a motioncompensation unit 710, an illumination compensation unit 720, anillumination difference prediction unit 730, a view synthesis predictionunit 740, and/or the like.

The NAL parser 100 carries out parsing by NAL units to decode a receivedvideo sequence. In general, at least one sequence parameter set and atleast one picture parameter set are transferred to a decoder before aslice header and slice data are decoded. In this case, various kinds ofconfiguration information may be included in a NAL header area or anextension area of a NAL header. Since MVC is an amendment technique fora conventional AVC technique, it may be more efficient to only add theconfiguration information if the bit stream is an MVC bit stream, ratherthan unconditional addition. For instance, flag information foridentifying a presence or non-presence of an MVC bit stream may be addedat the encoder in the NAL header area or the extension area of the NALheader. Only if an inputted bit stream is a multi-view video coded bitstream according to the flag information, is the addition ofconfiguration information for a multi-view video permitted. Forinstance, the configuration information may include temporal levelinformation, view level information, inter-view picture groupidentification information, view identification information, and/or thelike. This is explained in detail with reference to FIG. 2 as follows.

FIG. 2 is a diagram of configuration information for a multi-view videoaddable to a multi-view video coded bit stream according to oneembodiment of the present invention. Details of configurationinformation for a multi-view video are explained in the followingdescription.

Temporal level information may indicate information for a hierarchicalstructure to provide temporal scalability from a video signal ({circlearound (1)}). Through the temporal level information, it is possible toprovide a user with sequences on various time zones.

View level information indicates information for a hierarchicalstructure to provide view scalability from a video signal ({circlearound (2)}). In a multi-view video, it is desirable to define a levelfor a time and a level for a view to provide a user with varioustemporal and view sequences. In defining the above level information,temporal scalability and view scalability may be used. Hence, a user isable to select a sequence at a specific time and view, or a selectedsequence may be restricted by a condition.

The level information may be set in various ways according to a specificcondition. For instance, the level information may be set differentlyaccording to camera location or camera alignment. And, the levelinformation may be determined by considering view dependency. Forinstance, a level for a view having an I-picture in an inter-viewpicture group is set to 0, a level for a view having a P-picture in theinter-view picture group is set to 1, and a level for a view having aB-picture in the inter-view picture group is set to 2. Moreover, thelevel information may be randomly set not based on a special condition.The view level information will be explained in detail with reference toFIG. 4 and FIG. 5 later.

Inter-view picture group identification information indicatesinformation for identifying whether a coded picture of a current NALunit is an inter-view picture group ({circle around (3)}). In this case,the inter-view picture group means a coded picture in which all slicesreference only slices with the same picture order count. For instance,the inter-view picture group means a coded picture that refers to slicesin a different view only without referring to slices in a current view.In a decoding process of a multi-view video, an inter-view random accessmay be desired. The inter-view picture group identification informationmay be used to realize an efficient random access. And, inter-viewreference information may be used for inter-view prediction. So,inter-view picture group identification information can be used toobtain the inter-view reference information. Moreover, the inter-viewpicture group identification information can be used to add referencepictures for inter-view prediction in constructing a reference picturelist. Besides, the inter-view picture group identification informationcan be used to manage the added reference pictures for the inter-viewprediction. For instance, the reference pictures may be classified intointer-view picture groups and non-inter-view picture groups, and theclassified reference pictures can then be marked that the referencepictures failing to be used for the inter-view prediction shall not beused. Meanwhile, the inter-view picture group identification informationis applicable to a hypothetical reference decoder. Details of theinter-view picture group identification information will be explainedwith reference to FIG. 6 later.

The view identification information means information for discriminatinga picture in a current view from a picture in a different view ({circlearound (4)}). In coding a video signal, POC (picture order count) or‘frame_num’ may be used to identify each picture. In case of amulti-view video sequence, inter-view prediction can be executed. So,identification information to discriminate a picture in a current viewfrom a picture in another view is desired. So, it is possible to defineview identification information “view_id” for identifying a view of apicture. The view identification information can be obtained from aheader area of a video signal. For instance, the header area can be aNAL header area, an extension area of a NAL header, or a slice headerarea. Information for a picture in a view different from that of acurrent picture is obtained using the view identification informationand it is possible to decode the video signal using the information ofthe picture in the different view. This will be described in greaterdetail below. The view identification information is applicable to anoverall encoding/decoding process of the video signal. And, the viewidentification information can be applied to multi-view video codingusing the ‘frame_num’ that considers a view instead of considering aspecific view identifier.

Meanwhile, the entropy decoding unit 200 carries out entropy decoding ona parsed bit stream, and a coefficient of each macroblock, a motionvector, and the like are then extracted. The inversequantization/inverse transform unit 300 obtains a transformedcoefficient value by multiplying a received quantized value by aconstant and then transforms the coefficient value inversely toreconstruct a pixel value. Using the reconstructed pixel value, theintra-prediction unit 400 performs an intra prediction from a decodedsample within a current picture. Meanwhile, the deblocking filter unit500 is applied to each coded macroblock to reduce block distortion. Afilter smoothes a block edge to enhance an image quality of a decodedframe. A selection of a filtering process depends on boundary strengthand gradient of an image sample around a boundary. Pictures throughfiltering are outputted or stored in the decoded picture buffer unit 600to be used as reference pictures.

The decoded picture buffer unit 600 plays a role in storing or openingthe previously coded pictures to perform an inter prediction. In thiscase, to store the pictures in the decoded picture buffer unit 600 or toopen the pictures, ‘frame_num’ and POC (picture order count) of eachpicture are used. So, since there exist pictures in a view differentfrom that of a current picture among the previously coded pictures, viewinformation for identifying a view of a picture may be usable togetherwith the ‘frame_num’ and the POC. The decoded picture buffer unit 600includes the reference picture storing unit 610, the reference picturelist constructing unit 620, and/or the reference picture managing unit650. The reference picture storing unit 610 may store pictures that willbe referred to for the coding of the current picture. The referencepicture list constructing unit 620 may construct a list of referencepictures for the inter-picture prediction. In multi-view video coding,inter-view prediction may be desired. So, if a current picture refers toa picture in another view, it may be possible to construct a referencepicture list for inter-view prediction. In this case, the referencepicture list constructing unit 620 can use information for view ingenerating the reference picture list for the inter-view prediction.Details of the reference picture list constructing unit 620 will beexplained with reference to FIG. 3 later.

FIG. 3 is an internal block diagram of a reference picture listconstructing unit 620 according to an embodiment of the presentinvention.

The reference picture list constructing unit 620 includes the variablederiving unit 625, the reference picture list initializing unit 630, andthe reference list reordering unit 640.

The variable deriving unit 625 derives variables used for referencepicture list initialization. For instance, the variable may be derivedusing ‘frame_num’ indicating a picture identification number. Inparticular, variables FrameNum and FrameNumWrap may be usable for eachshort-term reference picture. First of all, the variable FrameNum isequal to a value of the syntax element frame_num discussed above. Thevariable FrameNumWrap may be used for the decoded picture buffer unit600 to assign a small number to each reference picture. And, thevariable FrameNumWrap may be derived from the variable FrameNum. Avariable PicNum may be derived using the derived variable FrameNumWrap.A variable PicNum may mean an identification number of a picture used bythe decoded picture buffer unit 600. In case of indicating a long-termreference picture, a variable LongTermPicNum can be usable.In order to construct a reference picture list for inter-viewprediction, a first variable (e.g., ViewNum) may be derived to constructa reference picture list for inter-view prediction. For instance, asecond variable (e.g., ViewId) may be derived using ‘view_id’ foridentifying a view of a picture. First of all, the second variableViewId may be equal to a value of the syntax element ‘view_id’. And, athird variable (e.g., ViewIdWrap) can be used for the decoded picturebuffer unit 600 to assign a small view identification number to eachreference picture and can be derived from the second variable. A firstvariable ViewNum may mean a view identification number of picture usedby the decoded picture buffer unit 600. Yet, since a number of referencepictures used for inter-view prediction in multi-view video coding maybe relatively smaller than that used for temporal prediction, anothervariable to indicate a view identification number of a long-termreference picture may not be defined.

The reference picture list initializing unit 630 initializes a referencepicture list using the above-mentioned variables. In this case, aninitialization process for the reference picture list may differaccording to a slice type. For instance, in case of decoding a P-slice,a reference index based on a decoding order may be assigned. In case ofdecoding a B-slice, a reference index based on a picture output ordermay be assigned. In case of initializing a reference picture list forinter-view prediction, an index may be assigned to a reference picturebased on the first variable ViewNum, i.e., the variable derived fromview information.

The reference picture list reordering unit 640 plays a role in enhancinga compression efficiency by assigning a smaller index to a picturefrequently referred to in the initialized reference picture list. Thisis because a small bit is assigned if a reference index for encodinggets smaller.

As shown in FIG. 12, the reference picture list reordering unit 640includes a slice type checking unit 642, a reference picture list-0reordering unit 643, and/or a reference picture list-1 reordering unit645. If an initialized reference picture list is inputted, the slicetype checking unit 642 checks a type of a slice to be decoded and thendecides whether to reorder a reference picture list-0 or a referencepicture list-1. So, the reference picture list-0/1 reordering unit643,645 performs reordering of the reference picture list-0 if the slicetype is not an I-slice and also performs reordering of the referencepicture list-1 additionally if the slice type is a B-slice. Thus, afteran end of the reordering process, a reference picture list isconstructed.

The reference picture list 0/1 reordering units 643, 645 includes anidentification information obtaining unit 643A,645A and a referenceindex assignment changing unit 643B,645B respectively. Theidentification information obtaining unit 643A,645A receivesidentification information (reordering_of_pic_nums_idc) indicating anassigning method of a reference index if reordering of a referencepicture list is carried out according to flag information (e.g., set bythe encoder and included in the header syntax) indicating whether toexecute the reordering of the reference picture list. And, the referenceindex assignment changing unit 643B,645B reorders the reference picturelist by changing an assignment of a reference index according to theidentification information.

And, the reference picture list reordering unit 640 is operable byanother method. For instance, reordering can be executed by checking aNAL unit type transferred prior to passing through the slice typechecking unit 642 and then classifying the NAL unit type into a case ofMVC NAL and a case of non-MVC NAL.

Returning to FIG. 1, the reference picture managing unit 650 managesreference pictures to execute inter prediction more flexibly. Forinstance, a memory management control operation method and a slidingwindow method are usable. This is to manage a reference picture memoryand a non-reference picture memory by unifying the memories into onememory and realize efficient memory management with a small memory. Inmulti-view video coding, since pictures in a view direction have thesame picture order count, information for identifying a view of each ofthe pictures may be usable in marking the pictures in a view direction.And, reference pictures managed in the above manner can be used by theinter-prediction unit 700.

The inter-prediction unit 700 carries out inter prediction usingreference pictures stored in the decoded picture buffer unit 600. Aninter-coded macroblock can be divided into macroblock partitions. And,each of the macroblock partitions can be predicted from one or tworeference pictures. The inter-prediction unit 700 includes the motioncompensation unit 710, the illumination compensation unit 720, theillumination difference prediction unit 730, the view synthesisprediction unit 740, the weighted prediction unit 750, and/or the like.

The motion compensation unit 710 compensates for a motion of a currentblock using information transferred from the entropy decoding unit 200.Motion vectors of neighboring blocks of the current block are extractedfrom a video signal, and then a motion vector predictor of the currentblock is derived from the motion vectors of the neighboring blocks. And,the motion of the current block is compensated using the derived motionvector predictor and a differential motion vector extracted from thevideo signal. And, motion compensation may be performed using onereference picture or a plurality of pictures. In multi-view videocoding, in case that a current picture refers to pictures in differentviews, motion compensation may be performed using reference picture listinformation for the inter-view prediction stored in the decoded picturebuffer unit 600. And, motion compensation may be performed using viewinformation for identifying a view of the reference picture.

A direct mode is a coding mode for predicting motion information of acurrent block from motion information for an encoded block. Since thismethod is able to save a number of bits required for coding the motioninformation, compression efficiency is enhanced. For instance, atemporal direct mode predicts motion information for a current blockusing a correlation of motion information in a temporal direction. Usinga method like this method, embodiments of the present invention maypredict motion information for a current block using correlation ofmotion information in a view direction.

Meanwhile, in case that an inputted bit stream corresponds to amulti-view video, since the respective view sequences are obtained bydifferent cameras, an illumination difference is generated by internaland external factors of the cameras. To prevent this, the illuminationcompensation unit 720 compensates the illumination difference. Inperforming the illumination compensation, flag information (e.g., set bythe encoder and included in the header syntax) indicating whether toperform illumination compensation on a specific layer of a video signalmay be used. For instance, an illumination compensation may be performedusing flag information indicating whether to perform the illuminationcompensation on a corresponding slice or macroblock. In performing theillumination compensation using the flag information, the illuminationcompensation is applicable to various macroblock types (e.g., inter16×16 mode, B-skip mode, direct mode, etc.).

In performing illumination compensation, information for a neighboringblock or information for a block in a view different from that of acurrent block may be used to reconstruct the current block. And, anillumination difference value of the current block may be used. In thiscase, if the current block refers to blocks in a different view,illumination compensation may be performed using the reference picturelist information for the inter-view prediction stored in the decodedpicture buffer unit 600. In this case, the illumination difference valueof the current block indicates a difference between an average pixelvalue of the current block and an average pixel value of a referenceblock corresponding to the current block. For the example of using theillumination difference value, the illumination difference predictionvalue of the current block is obtained using neighboring blocks of thecurrent block, and a difference value (illumination difference residual)between the illumination difference value and the illuminationdifference prediction value is used. Hence, the decoding unit is able toreconstruct the illumination difference value of the current block usingthe illumination difference residual and the illumination differenceprediction value. In obtaining an illumination difference predictionvalue of a current block, information for a neighboring block may beused. For instance, an illumination difference value of a current blockmay be predicted using an illumination difference value of a neighborblock. Prior to the prediction, it may be checked whether a referenceindex of the current block is equal to that of the neighboring block.According to a result of the checking, it is then decided what kind of aneighboring block or a value will be used.

The view synthesis prediction unit 740 is used to synthesize pictures ina virtual view using pictures in a view neighboring a view of a currentpicture and to predict the current picture using the synthesizedpictures in the virtual view. The decoding unit is able to decidewhether to synthesize a picture in a virtual view according to aninter-view synthesis prediction identifier (view_synthesis_pred_flag orview_syn_pred_flag) transferred from an encoding unit. For instance, ifthe view_synthesize_pred_flag=1 or view_syn_pred_flag=1, a slice ormacroblock in a virtual view is synthesized. In this case, when theinter-view synthesis prediction identifier informs that a virtual viewwill be generated, a picture in the virtual view may be generated usingview information for identifying a view of a picture. And, in predictinga current picture from the synthesized pictures in the virtual view, theview information may be used to use the picture in the virtual view as areference picture.

The weighted prediction unit 750 is used to compensate for a phenomenonthat an image quality of a sequence is considerably degraded in case ofencoding the sequence of which brightness temporarily varies. In MVC,weighted prediction may be performed to compensate for a brightnessdifference from a sequence in a different view as well as for a sequenceof which brightness temporarily varies. For instance, the weightedprediction method can be classified into explicit weighted predictionmethod and implicit weighted prediction method.

In particular, the explicit weighted prediction method can use onereference picture or two reference pictures. In case of using onereference picture, a prediction signal is generated from multiplying aprediction signal corresponding to motion compensation by a weightcoefficient. In case of using two reference pictures, a predictionsignal is generated from adding an offset value to a value resultingfrom multiplying a prediction signal corresponding to motioncompensation by a weight coefficient.

And, the implicit weighted prediction performs a weighted predictionusing a distance from a reference picture. As a method of obtaining thedistance from the reference picture, the POC (picture order count)indicating a picture output order, for example, may be used. In thiscase, the POC may be obtained by considering identification of a view ofeach picture. In obtaining a weight coefficient for a picture in adifferent view, view information for identifying a view of a picture maybe used to obtain a distance between views of the respective pictures.

In video signal coding, depth information is usable for a specificapplication or another purpose. In this case, the depth information maymean information capable of indicating an inter-view disparitydifference. For instance, a disparity vector may be obtained byinter-view prediction. And, the obtained disparity vector should betransferred to a decoding apparatus for disparity compensation of acurrent block. Yet, if a depth map is obtained and then transferred tothe decoding apparatus, the disparity vector can be inferred from thedepth map (or disparity map) without transferring the disparity vectorto the decoding apparatus. In this case, it is advantageous in that thenumber of bits of depth information to be transferred to the decodingapparatus can be reduced. So, by deriving the disparity vector from thedepth map, a new disparity compensating method may be provided. Thus, incase of using a picture in a different view in the course of derivingthe disparity vector from the depth map, view information foridentifying a view of a picture can be used.

The inter-predicted or intra-predicted pictures through theabove-explained process are selected according to a prediction mode toreconstruct a current picture. In the following description, variousembodiments providing an efficient decoding method of a video signal areexplained in detail.

FIG. 4 is a diagram of a hierarchical structure of level information forproviding view scalability of a video signal according to one embodimentof the present invention.

Referring to FIG. 4, level information for each view can be decided byconsidering inter-view reference information. For instance, since it isimpossible to decode a P-picture and a B-picture without an I-picture, a‘level=0’ may be assigned to a base view of which inter-view picturegroup is the I-picture, ‘level=1’ may be assigned to a base view ofwhich inter-view picture group is the P-picture, and ‘level=2’ may beassigned to a base view of which inter-view picture group is theB-picture. Yet, level information may be decided randomly according to aspecific standard.

Level information can be randomly decided according to a specificstandard or without a standard. For instance, in case that levelinformation is decided based on a view, a view V0 as a base view may beset to view level 0, a view of pictures predicted using pictures in oneview may be set to view level 1, and a view of pictures predicted usingpictures in a plurality of views may be set to view level 2. In thiscase, at least one view sequence to have compatibility with aconventional decoder (e.g., H.264/AVC, MPEG-2, MPEG-4, etc.) may beneeded. This base view becomes a base of multi-view coding, which maycorrespond to a reference view for prediction of another view. Asequence corresponding to a base view in MVC (multi-view video coding)can be configured into an independent bit stream by being encoded by aconventional sequence encoding scheme (MPEG-2, MPEG-4, H.263, H.264,etc.). A sequence corresponding to a base view may be compatible withH.264/AVC or not. Yet, a sequence in a view compatible with H.264/AVCcorresponds to a base view.

As can be seen in FIG. 4, a view V2 of pictures may be predicted usingpictures in the view V0 a view V4 of pictures may be predicted usingpictures in the view V2, a view V6 of pictures may be predicted usingpictures in the view V4, and a view V7 of pictures may be predictedusing pictures in the view V6 to view level 1. And, a view V1 ofpictures may be predicted using pictures in the views V0 and V2 and aview V3 may be predicted in the same manner, and a view V5 may bepredicted in the same manner to view level 2. So, in case that a user'sdecoder is unable to view a multi-view video sequence, it decodessequences in the view corresponding to the view level 0 only. In casethat the user's decoder is restricted by profile information, may decodethe information of a restricted view level only. In this case, a profilemeans that technical elements for an algorithm in a videoencoding/decoding process are standardized. In particular, the profileis a set of technical elements required for decoding a bit sequence of acompressed sequence and can be a sort of a sub- or component-standard.

According to another embodiment of the present invention, levelinformation may vary according to a location of a camera. For instance,assuming that views V0 and V1 are sequences obtained by a camera locatedin front, that views V2 and V3 are sequences obtained by a cameralocated in rear, that views V4 and V5 are sequences obtained by a cameralocated to the left, and that views V6 and V7 are sequences obtained bya camera located to the right, the views V0 and V1 may be set to viewlevel 0, the views V2 and V3 may be set to view level 1, the views V4and V5 may be set to view level 2, and the views V6 and V7 may be set toview level 3. Alternatively, level information may vary according tocamera alignment. Alternatively, level information can be randomlydecided not based on a specific standard.

FIGS. 2 and 5 are diagrams of a NAL-unit configuration including levelinformation within an extension area of a NAL header according to oneembodiment of the present invention.

Referring to FIGS. 2 and 5, a NAL unit basically includes a NAL headerand an RBSP. The NAL header includes flag information (nal_ref_idc)indicating whether a slice becoming a reference picture of the NAL unitis included and an identifier (nal_unit_type) indicating a type of theNAL unit. And, the NAL header may further include level information(view_level) indicating information for a hierarchical structure toprovide view scalability.

Compressed original data is stored in the RBSP, and RBSP trailing bit orbits are added to a last portion of the RBSP to represent a length ofthe RBSP as an 8-bit multiplication number. As the types of the NALunit, there are IDR (instantaneous decoding refresh), SPS (sequenceparameter set), PPS (picture parameter set), SEI (supplementalenhancement information), etc.

Referring to FIG. 5 the NAL header includes information for a viewidentifier. And, a video sequence of a corresponding view level isdecoded with reference to the view identifier in the course ofperforming decoding according to a view level.

The NAL unit includes a NAL header 51 and a slice layer 53. The NALheader 51 includes a NAL header extension 52. And, the slice layer 53includes a slice header 54 and a slice data 55 as the RBSP.

The NAL header 51 includes an identifier (nal_unit_type) indicating atype of the NAL unit (See FIG. 2 also). For instance, the identifierindicates the NAL unit type may be an identifier for both scalablecoding and multi-view video coding. In this case, the NAL headerextension 52 can include flag information discriminating whether acurrent NAL is the NAL for the scalable video coding or the NAL for themulti-view video coding. And, the NAL header extension 52 can includeextension information for the current NAL according to the flaginformation. For instance, in case that the current NAL is the NAL forthe multi-view video coding according to the flag information, the NALheader extension 52 can include level information (view_level)indicating information for a hierarchical structure to provide viewscalability.

FIG. 6 is a diagram of an overall predictive structure of a multi-viewvideo signal according to one embodiment of the present invention toexplain a concept of an inter-view picture group.

Referring to FIG. 6, T0 to T100 on a horizontal axis indicate framesaccording to time and S0 to S7 on a vertical axis indicate framesaccording to view. For instance, pictures at T0 mean frames captured bydifferent cameras at the same time zone T0, while pictures at S0 meansequences captured by a single camera at different time zones. And,arrows in the drawing indicate predictive directions and predictiveorders of the respective pictures. For instance, a picture P0 in a viewS2 at a time zone T0 is a picture predicted from I0, which becomes areference picture of a picture P0 in a view S4 at the time zone T0. And,it becomes a reference picture of pictures B1 and B2 at time zones T4and T2 in the view S2, respectively.

In a multi-view video decoding process, an inter-view random access maybe needed. So, an access to a random view should be possible byminimizing the decoding effort. In this case, a concept of an inter-viewpicture group may be needed to realize an efficient access. Theinter-view picture group means a coded picture in which all slicesreference only slices with the same picture order count. For instance,the inter-view picture group means a coded picture that refers to slicesin a different view only without referring to slices in a current view.In FIG. 6, if a picture I0 in a view S0 at a time zone T0 is aninter-view picture group, all pictures in different views on the sametime zone, i.e., the time zone T0, become inter-view picture groups. Foranother instance, if a picture I0 in a view S0 at a time zone T8 is aninter-view picture group, all pictures in different views at the sametime zone, i.e., the time zone T8, are inter-view picture groups.Likewise, all pictures in T16, . . . , T96, and T100 become inter-viewpicture groups as well.

FIG. 7 is a diagram of a predictive structure according to an embodimentof the present invention to explain a concept of a newly definedinter-view picture group.

In an overall predictive structure of MVC, GOP can begin with anI-picture. And, the I-picture is compatible with H.264/AVC. So, allinter-view picture groups compatible with H.264/AVC can always becomethe I-picture. Yet, in case that the I-pictures are replaced by aP-picture, more efficient coding is enabled. In particular, moreefficient coding is enabled using the predictive structure enabling GOPto begin with the P-picture compatible with H.264/AVC.

In this case, if the inter-view picture group is re-defined, all slicesbecome encoded pictures capable of referring to not only a slice in aframe on a same time zone but also to a slice in the same view on adifferent time zone. Yet, in case of referring to a slice on a differenttime zone in a same view, it can be restricted to the inter-view picturegroup compatible with H.264/AVC only. For instance, a P-picture on atiming point T8 in a view S0 in FIG. 6 can become a newly definedinter-view picture group. Likewise, a P-picture on a timing point T96 ina view S0 or a P-picture on a timing point T100 in a view S0 can becomea newly defined inter-view picture group. And, the inter-view picturegroup can be defined only if it is a base view.

After the inter-view picture group has been decoded, all of thesequentially coded pictures are decoded from pictures decoded ahead ofthe inter-view picture group in an output order withoutinter-prediction.

Considering the overall coding structure of the multi-view video shownin FIG. 6 and FIG. 7, since inter-view reference information of aninter-view picture group differs from that of a non-inter-view picturegroup, the inter-view picture group and the non-inter-view picture groupmay be discriminated from each other according to the inter-view picturegroup identification information.

The inter-view reference information means the information capable ofrecognizing a predictive structure between inter-view pictures. This canbe obtained from a data area of a video signal. For instance, it can beobtained from a sequence parameter set area. And, the inter-viewreference information can be recognized using the number of referencepictures and view information for the reference pictures. For instance,the number of total views is obtained and the view information foridentifying each view can be then obtained based on the number of thetotal views. And, the number of the reference pictures for a referencedirection for each view may be obtained. According to the number of thereference pictures, the view information for each of the referencepictures may be obtained. In this manner, the inter-view referenceinformation can be obtained. And, the inter-view reference informationcan be recognized by discriminating an inter-view picture group and anon-inter-view picture group. This can be recognized using inter-viewpicture group identification information indicating whether a codedslice in a current NAL is an inter-view picture group. Details of theinter-view picture group identification information are explained withreference to FIG. 8 as follows.

FIG. 8 is a schematic block diagram of an apparatus for decoding amulti-view video using inter-view picture group identifying informationaccording to one embodiment of the present invention.

Referring to FIG. 8, a decoding apparatus according to one embodiment ofthe present invention includes a bit stream deciding unit 81, aninter-view picture group identification information obtaining unit 82,and a multi-view video decoding unit 83.

If a bit stream is inputted, the bit stream deciding unit 81 decideswhether the inputted bit stream is a coded bit stream for a scalablevideo coding or a coded bit stream for multi-view video coding. This canbe decided by flag information included in the bit stream.

The inter-view picture group identification information obtaining unit82 is able to obtain inter-view picture group identification informationif the inputted bit stream is the bit stream for a multi-view videocoding as a result of the decision. If the obtained inter-view picturegroup identification information is ‘true’, it means that a coded sliceof a current NAL is an inter-view picture group. If the obtainedinter-view picture group identification information is ‘false’, it meansthat a coded slice of a current NAL is a non-inter-view picture group.The inter-view picture group identification information can be obtainedfrom an extension area of a NAL header or a slice layer area.

The multi-view video decoding unit 83 decodes a multi-view videoaccording to the inter-view picture group identification information.According to an overall coding structure of a multi-view video sequence,inter-view reference information of an inter-view picture group differsfrom that of a non-inter-view picture group. So, the inter-view picturegroup identification information may be used in adding referencepictures for inter-view prediction to generate a reference picture list,for example. And, the inter-view picture group identificationinformation may be used to manage the reference pictures for theinter-view prediction. Moreover, the inter-view picture groupidentification information is applicable to a hypothetical referencedecoder.

As another example of using the inter-view picture group identificationinformation, in case of using information in a different view for eachdecoding process, inter-view reference information included in asequence parameter set is usable. In this case, information fordiscriminating whether a current picture is an inter-view picture groupor a non-inter-view picture group, i.e., inter-view picture groupidentification information may be required. So, it is able to usedifferent inter-view reference information for each decoding process.

FIG. 9 is a flowchart of a process for generating a reference picturelist according to an embodiment of the present invention.

Referring to FIG. 9, the decoded picture buffer unit 600 plays a role instoring or opening previously coded pictures to perform inter-pictureprediction.

First of all, pictures coded prior to a current picture are stored inthe reference picture storing unit 610 to be used as reference pictures(S91).

In multi-view video coding, since some of the previously coded picturesare in a view different from that of the current picture, viewinformation for identifying a view of a picture can be used to utilizethese pictures as reference pictures. So, the decoder should obtain viewinformation for identifying a view of a picture (S92). For instance, theobtained view information can include ‘view_id’ for identifying a viewof a picture.

The decoded picture buffer unit 600 derives a variable used therein togenerate a reference picture list. Since inter-view prediction may berequired for multi-view video coding, if a current picture refers to apicture in a different view, a reference picture list may be generatedfor inter-view prediction. In this case, the decoded picture buffer unit600 derives a variable used to generate the reference picture list forthe inter-view prediction using the obtained view information (S93).

A reference picture list for temporal prediction or a reference picturelist for inter-view prediction can be generated by different methodsaccording to a slice type of a current slice (S94). For instance, if aslice type is a P/SP slice, a reference picture list 0 is generated(S95). In case that a slice type is a B-slice, a reference picture list0 and a reference picture list 1 are generated (S96). In this case, thereference picture list 0 or 1 can include the reference picture list forthe temporal prediction only or both of the reference picture list forthe temporal prediction and the reference picture list for theinter-view prediction. This will be explained in detail with referenceto FIG. 8 and FIG. 9 later.

The initialized reference picture list undergoes a process for assigninga smaller number to a frequently referred to pictures to further enhancea compression rate (S97). And, this can be called a reordering processfor a reference picture list, which will be explained in detail withreference to FIGS. 12 to 19 later. The current picture is decoded usingthe reordered reference picture list and the decoded picture buffer unit600 manages the decoded reference pictures to operate a buffer moreefficiently (S98). The reference pictures managed by the above processare read by the inter-prediction unit 700 to be used forinter-prediction. In multi-view video coding, the inter-prediction caninclude inter-view prediction. In this case, the reference picture listfor the inter-view prediction is usable.

Detailed examples for a method of generating a reference picture listaccording to a slice type are explained with reference to FIG. 10 andFIG. 11 as follows.

FIG. 10 is a diagram to explain a method of initializing a referencepicture list when a current slice is a P-slice according to oneembodiment of the present invention.

Referring to FIG. 10, a time is indicated by T0, T1, . . . , TN, while aview is indicated by V0, V1, . . . , V4. For instance, a current pictureindicates a picture at a time T3 in a view V4. And, a slice type of thecurrent picture is a P-slice. ‘PN’ is an abbreviation of a variablePicNum, ‘LPN’ is an abbreviation of a variable LongTermPicNum, and ‘VN’is an abbreviation of a variable ViewNum. A numeral attached to an endportion of each of the variables indicates an index indicating a time ofeach picture (for PN or LPN) or a view of each picture (for VN). This isapplicable to FIG. 11 in the same manner.

A reference picture list for temporal prediction or a reference picturelist for inter-view prediction can be generated in a different wayaccording to a slice type of a current slice. For instance, a slice typein FIG. 10 is a P/SP slice. In this case, a reference picture list 0 isgenerated. In particular, the reference picture list 0 can include areference picture list for temporal prediction and/or a referencepicture list for inter-view prediction. In the present embodiment, it isassumed that a reference picture list includes both a reference picturelist for temporal prediction and a reference picture list for inter-viewprediction.

There are various methods for ordering reference pictures. For instance,reference pictures can be aligned in order of decoding or pictureoutput. Alternatively, reference pictures can be aligned based on avariable derived using view information. Alternatively, referencepictures can be aligned according to inter-view reference informationindicating an inter-view prediction structure.

In case of a reference picture list for temporal prediction, short-termreference pictures and long-term reference pictures can be aligned basedon a decoding order. For instance, they can be aligned according to avalue of a variable PicNum or LongTermPicNum derived from a valueindicating a picture identification number (e.g., frame_num orLongtermframeidx). First of all, short-term reference pictures can beinitialized prior to long-term reference pictures. An order of aligningthe short-term reference pictures can be set from a reference picturehaving a highest value of variable PicNum to a reference picture havinga lowest variable value. For instance, the short-term reference picturescan be aligned in order of PN1 having a highest variable, PN2 having anintermediate variable, and PNO having a lowest variable among PN0 toPN2. An order of aligning the long-term reference pictures can be setfrom a reference picture having a lowest value of variableLongTermPicNum to a reference picture having a highest variable value.For instance, the long-term reference pictures can be aligned in orderof LPN0 having a highest variable and LPN1 having a lowest variable.

In case of a reference picture list for inter-view prediction, referencepictures can be aligned based on a first variable ViewNum derived usingview information. In particular, reference pictures can be aligned inorder of a reference picture having a highest first variable (ViewNum)value to a reference picture having a lowest first variable (ViewNum)value. For instance, reference pictures can be aligned in order of VN3having a highest variable, VN2, VN1, and VN0 having a lowest variableamong VN0, VN1, VN2, and VN3.

Thus, the reference picture list for the temporal prediction and thereference picture list for the inter-view prediction can be managed asone reference picture list. Alternatively, the reference picture listfor the temporal prediction and the reference picture list for theinter-view prediction can be managed as separate reference picturelists, respectively. In case of managing the reference picture list forthe temporal prediction and the reference picture list for theinter-view prediction as one reference picture list, they can beinitialized according to an order or simultaneously. For instance, incase of initializing the reference picture list for the temporalprediction and the reference picture list for the inter-view predictionaccording to an order, the reference picture list for the temporalprediction may be initialized first and the reference picture list forthe inter-view prediction is then initialized. This concept isapplicable to FIG. 11 as well.

A case that a slice type of a current picture is a B-slice is explainedwith reference to FIG. 11 as follows.

FIG. 11 is a diagram to explain a method of initializing a referencepicture list when a current slice is a B-slice according to oneembodiment of the present invention.

Referring to FIG. 9, in case that a slice type is a B-slice, a referencepicture list 0 and a reference picture list 1 are generated. In thiscase, the reference picture list 0 or the reference picture list 1 caninclude a reference picture list for temporal prediction only or both areference picture list for temporal prediction and a reference picturelist for inter-view prediction.

In case of the reference picture list for temporal prediction, ashort-term reference picture aligning method may differ from a long-termreference picture aligning method. For instance, in case of short-termreference pictures, reference pictures can be aligned according to apicture order count (hereinafter abbreviated POC). In case of long-termreference pictures, reference pictures can be aligned according to avariable (LongtermPicNum) value. And, the short-term reference picturescan be initialized prior to the long-term reference pictures.

In order of aligning short-term reference pictures of the referencepicture list 0, reference pictures may be aligned from a referencepicture having a highest POC value to a reference picture having alowest POC value among reference pictures having POC values smaller thanthat of a current picture, and then aligned from a reference picturehaving a lowest POC value to a reference picture having a highest POCvalue among reference pictures having POC values greater than that ofthe current picture. For instance, reference pictures can be alignedfrom PN1, having a highest POC value out of reference pictures PN0 andPN1 having POC values smaller than that of a current picture, to PN0.Then, reference pictures may be aligned from PN3, having a lowest POCvalue out of reference pictures PN3 and PN4 having a POC value smallerthan that of a current picture, to PN4.

In order of aligning long-term reference pictures of the referencepicture list 0, reference pictures are aligned from a reference picturehaving a lowest variable LongtermPicNum to a reference picture having ahighest variable. For instance, reference pictures are aligned fromLPN0, having a lowest value out of LPN0 and LPN1, to LPN1 having asecond lowest variable.

In case of the reference picture list for the inter-view prediction,reference pictures can be aligned based on a first variable ViewNumderived using view information. For instance, in case of the referencepicture list 0 for the inter-view prediction, reference pictures can bealigned from a reference picture having a highest first variable valueamong reference pictures having first variable values lower than that ofa current picture to a reference picture having a lowest first variablevalue. The reference pictures are then aligned from a reference picturehaving a lowest first variable value among reference pictures havingfirst variable values greater than that of the current picture to areference picture having a highest first variable value. For instance,reference pictures may be aligned from VN1, having a highest firstvariable value out of VN0 and VN1 having first variable values smallerthan that of a current picture, to VN0 having a lowest first variablevalue. Then aligned from VN3, having a lowest first variable value outof VN3 and VN4 having first variable values greater than that of thecurrent picture, to VN4 having a highest first variable value.

In case of the reference picture list 1, the above-explained aligningmethod of the reference list 0 is similarly applicable to referencepicture list 1.

First of all, in case of the reference picture list for the temporalprediction, in order of aligning short-term reference pictures of thereference picture list 1, reference pictures may be aligned from areference picture having a lowest POC value to a reference picturehaving a highest POC value among reference pictures having POC valuesgreater than that of a current picture. Then the reference pictures maybe aligned from a reference picture having a highest POC value to areference picture having a lowest POC value among reference pictureshaving POC values smaller than that of the current picture. Forinstance, reference pictures may be aligned from PN3, having a lowestPOC value out of reference pictures PN3 and PN4 having POC valuesgreater than that of a current picture, to PN4. Then the referencepicture may be aligned from PN1, having a highest POC value out ofreference pictures PN0 and PN1 having POC values greater than that ofthe current picture, to PN0.

In order of aligning long-term reference pictures of the referencepicture list 1, reference pictures are aligned from a reference picturehaving a lowest variable LongtermPicNum to a reference picture having ahighest variable. For instance, reference pictures are aligned fromLPN0, having a lowest value out of LPN0 and LPN1, to LPN1 having alowest variable.

In case of the reference picture list for the inter-view prediction,reference pictures can be aligned based on a first variable ViewNumderived using view information. For instance, in case of the referencepicture list 1 for the inter-view prediction, reference pictures can bealigned from a reference picture having a lowest first variable valueamong reference pictures having first variable values greater than thatof a current picture to a reference picture having a highest firstvariable value. The reference pictures may then be aligned from areference picture having a highest first variable value among referencepictures having first variable values smaller than that of the currentpicture to a reference picture having a lowest first variable value. Forinstance, reference pictures may be aligned from VN3, having a lowestfirst variable value out of VN3 and VN4 having first variable valuesgreater than that of a current picture, to VN4 having a highest firstvariable value. Then the reference pictures may be aligned from VN1,having a highest first variable value out of VN0 and VN1 having firstvariable values smaller than that of the current picture, to VN0 havinga lowest first variable value.

The reference picture list initialized by the above process istransferred to the reference picture list reordering unit 640. Theinitialized reference picture list is then reordered for more efficientcoding. The reordering process is to reduce a bit rate by assigning asmall number to a reference picture having highest probability of beingselected as a reference picture. Various methods of reordering areference picture list are explained with reference to FIGS. 12 to 19 asfollows.

FIG. 12 is an internal block diagram of the reference picture listreordering unit 640 according to one embodiment of the presentinvention.

Referring to FIG. 12, the reference picture list reordering unit 640basically includes a slice type checking unit 642, a reference picturelist 0 reordering unit 643, and/or a reference picture list 1 reorderingunit 645.

In particular, the reference picture list 0 reordering unit 643 includesa first identification information obtaining unit 643A, and a firstreference index assignment changing unit 643B. And, the referencepicture list 1 reordering unit 645 includes a second identificationobtaining unit 645A and a second reference index assignment changingunit 645B.

The slice type checking unit 642 checks a slice type of a current slice.It is then decided whether to reorder a reference picture list 0 and/ora reference picture list 1 according to the slice type. For instance, ifa slice type of a current slice is an I-slice, both the referencepicture list 0 and the reference picture list 1 are not reordered. If aslice type of a current slice is a P-slice, the reference picture list 0is reordered only. If a slice type of a current slice is a B-slice, boththe reference picture list 0 and the reference picture list 1 arereordered.

The reference picture list 0 reordering unit 643 is activated if flaginformation (ref_pic_list_reordering_flag_l0 orref_view_list_reordering_flag_l0 of FIG. 17), received from the encoder,for executing reordering of the reference picture list 0 is ‘true’ andif the slice type of the current slice is not the I-slice. The firstidentification information obtaining unit 643A obtains identificationinformation (reordering_of_pic_nums_idc in FIG. 17) indicating areference index assigning method. The first reference index assignmentchanging unit 643B changes a reference index assigned to each referencepicture of the reference picture list 0 according to the identificationinformation.

Likewise, the reference picture list 1 reordering unit 645 is activatedif flag information (ref_pic_list_reordering_flag_l1 orref_view_list_reordering_flag_l1 of FIG. 17), received from the encoder,for executing reordering of the reference picture list 1 is ‘true’ andif the slice type of the current slice is the B-slice. The secondidentification information obtaining unit 645A obtains identificationinformation (reordering_of_pic_nums_idc of FIG. 17) indicating areference index assigning method. The second reference index assignmentchanging unit 645B changes a reference index assigned to each referencepicture of the reference picture list 1 according to the identificationinformation.

So, reference picture list information used for actual inter-predictionis generated through the reference picture list 0 reordering unit 643and the reference picture list 1 reordering unit 645.

A method of changing a reference index assigned to each referencepicture by the first or second reference index assignment changing unit643B or 645B is explained with reference to FIG. 13 as follows.

FIG. 13 is an internal block diagram of a reference index assignmentchanging unit 643B or 645B according to one embodiment of the presentinvention. In the following description, the reference picture list 0reordering unit 643 and the reference picture list 1 reordering unit 645shown in FIG. 12 are explained together. Furthermore, for FIGS. 12 and13, the flags and identification information may be the same as oranalogous to that described in detail with respect to FIGS. 15-19.

Referring to FIG. 13, each of the first and second reference indexassignment changing units 643B and 645B includes a reference indexassignment changing unit for temporal prediction 644A, a reference indexassignment changing unit for long-term reference picture 644B, areference index assignment changing unit for inter-view prediction 644C,and a reference index assignment change terminating unit 644D. Accordingto identification information obtained by the first and secondidentification information obtaining units 643A and 645A, parts withinthe first and second reference index assignment changing units 643B and645B are activated, respectively. And, the reordering process keepsbeing executed until identification information for terminating thereference index assignment change is inputted.

For instance, if identification information for changing assignment of areference index for temporal prediction is received from the first orsecond identification information obtaining unit 643A or 645A, thereference index assignment changing unit for temporal prediction 644A isactivated. The reference index assignment changing unit for temporalprediction 644A obtains a picture number difference according to thereceived identification information. In this case, the picture numberdifference means a difference between a picture number of a currentpicture and a predicted picture number. And, the predicted picturenumber may indicate a number of a reference picture assigned rightbefore. So, it is able to change the assignment of the reference indexusing the obtained picture number difference. In this case, the picturenumber difference can be added/subtracted to/from the predicted picturenumber according to the identification information.

For another instance, if identification information for changingassignment of a reference index to a designated long-term referencepicture is received, the reference index assignment changing unit for along-term reference picture 644B is activated. The reference indexassignment changing unit for a long-term reference picture 644B obtainsa long-term reference picture number of a designated picture accordingto the identification number.

For another instance, if identification information for changingassignment of a reference index for inter-view prediction is received,the reference index assignment changing unit for inter-view prediction644C is activated. The reference index assignment changing unit forinter-view prediction 644C obtains a view information differenceaccording to the identification information. In this case, the viewinformation difference means a difference between a view number of acurrent picture and a predicted view number. And, the predicted viewnumber may indicate a view number of a reference picture assigned rightbefore. So, it is able to change assignment of a reference index usingthe obtained view information difference. In this case, the viewinformation difference can be added/subtracted to/from the predictedview number according to the identification information.

For another instance, if identification information for terminating areference index assignment change is received, the reference indexassignment change terminating unit 644D is activated. The referenceindex assignment change terminating unit 644D terminates an assignmentchange of a reference index according to the received identificationinformation. So, the reference picture list reordering unit 640generates reference picture list information.

Thus, reference pictures used for inter-view prediction can be managedtogether with reference pictures used for temporal prediction.Alternatively, reference pictures used for inter-view prediction can bemanaged separate from reference pictures used for temporal prediction.For this, new information for managing the reference pictures used forthe inter-view prediction may be required. This will be explained withreference to FIGS. 15 to 19 later.

Details of the reference index assignment changing unit for inter-viewprediction 644C are explained with reference to FIG. 14 as follows.

FIG. 14 is a diagram to explain a process for reordering a referencepicture list using view information according to one embodiment of thepresent invention.

Referring to FIG. 14, for a view number VN of a current picture is 3, asize of a decoded picture buffer DPBsize is 4 and a slice type of acurrent slice is a P-slice, a reordering process for a reference picturelist 0 is explained as follows.

First of all, an initially predicted view number is ‘3’—that is the viewnumber of the current picture. And, an initial alignment of thereference picture list 0 for inter-view prediction is ‘4, 5, 6, 2’({circle around (1)}). In this case, if identification information forchanging assignment of a reference index for inter-view prediction bysubtracting a view information difference is received, ‘1’ is obtainedas the view information difference according to the receivedidentification information. For example, diff_view_num_minus1 in FIG. 18is 1. A newly predicted view number (=2) is calculated by subtractingthe view information difference (=1) from the predicted view number(=3). In particular, a first index of the reference picture list 0 forthe inter-view prediction is assigned to a reference picture having theview number 2. And, a picture previously assigned to the first index canbe moved to a most rear part of the reference picture list 0. So, thereordered reference picture list 0 is ‘2, 5, 6, 4’ ({circle around(2)}). Subsequently, if identification information for changingassignment of a reference index for inter-view prediction by subtractingthe view information difference is received, ‘−2’ is obtained as theview information difference according to the identification information.A newly predicted view number (=4) is then calculated by subtracting theview information difference (=−2) from the predicted view number (=2).In particular, a second index of the reference picture list 0 for theinter-view prediction is assigned to a reference picture having a viewnumber 4. Hence, the reordered reference picture list 0 is ‘2, 4, 6, 5’({circle around (3)}). Subsequently, if identification information forterminating the reference index assignment change is received, thereference picture list 0 having the reordered reference picture list 0as an end is generated according to the received identificationinformation ({circle around (4)}). Hence, the order of the finallygenerated reference picture list 0 for the inter-view prediction is ‘2,4, 6, 5’.

For another instance of reordering the rest of the pictures after thefirst index of the reference picture list 0 for the inter-viewprediction has been assigned, a picture assigned to each index can bemoved to a position right behind that of the corresponding picture. Inparticular, a second index is assigned to a picture having a view number4, a third index is assigned to a picture (view number 5) to which thesecond index was assigned, and a fourth index is assigned to a picture(view number 6) to which the third index was assigned. Hence, thereordered reference picture list 0 becomes ‘2, 4, 5, 6’. And, asubsequent reordering process can be executed in the same manner.

The reference picture list generated by the above-explained process isused for inter-prediction. Both the reference picture list for theinter-view prediction and the reference picture list for the temporalprediction can be managed as one reference picture list. Alternatively,each of the reference picture list for the inter-view prediction and thereference picture list for the temporal prediction can be managed as aseparate reference picture list. This is explained with reference toFIGS. 15 to 19 as follows.

FIG. 15 is an internal block diagram of a reference picture listreordering unit 640 according to another embodiment of the presentinvention.

Referring to FIG. 15, in order to manage a reference picture list forinter-view prediction as a separate reference picture list, newinformation may be provided. For instance, a reference picture list fortemporal prediction is reordered, and/or a reference picture list forinter-view prediction is then reordered in some cases.

The reference picture list reordering unit 640 basically includes areference picture list reordering unit for temporal prediction 910, aNAL type checking unit 960, and/or a reference picture list reorderingunit for inter-view prediction 970.

The reference picture list reordering unit for temporal prediction 910includes a slice type checking unit 642, a third identificationinformation obtaining unit 920, a third reference index assignmentchanging unit 930, a fourth identification information obtaining unit940, and/or a fourth reference index assignment changing unit 950. Thethird reference index assignment changing unit 930 includes a referenceindex assignment changing unit for temporal prediction 930A, a referenceindex assignment changing unit for a long-term reference picture 930B,and/or a reference index assignment change terminating unit 930C.Likewise, the fourth reference index assignment changing unit 950includes a reference index assignment changing unit for temporalprediction 950A, a reference index assignment changing unit forlong-term reference picture 950B, and/or a reference index assignmentchange terminating unit 950C.

The reference picture list reordering unit for temporal prediction 910reorders reference pictures used for temporal prediction. Operations ofthe reference picture list reordering unit for temporal prediction 910are identical to those of the aforesaid reference picture listreordering unit 640 shown in FIG. 10 except information for thereference pictures for the inter-view prediction. So, details of thereference picture list reordering unit for temporal prediction 910 areomitted in the following description.

The NAL type checking unit 960 checks a NAL type of a received bitstream. If the NAL type is a NAL for multi-view video coding, referencepictures used for the inter-view prediction are reordered by thereference picture list reordering unit for temporal prediction 970. Thegenerated reference picture list for the inter-view prediction are usedfor inter-prediction together with the reference picture list generatedby the reference picture list reordering unit for temporal prediction910. Yet, if the NAL type is not the NAL for the multi-view videocoding, the reference picture list for the inter-view prediction is notreordered. In this case, a reference picture list for temporalprediction is generated only. And, the inter-view prediction referencepicture list reordering unit 970 reorders reference pictures used forinter-view prediction. This is explained in detail with reference toFIG. 16 as follows.

FIG. 16 is an internal block diagram of the reference picture listreordering unit 970 for inter-view prediction according to oneembodiment of the present invention.

Referring to FIG. 16, the reference picture list reordering unit forinter-view prediction 970 includes a slice type checking unit 642, afifth identification information obtaining unit 971, a fifth referenceindex assignment changing unit 972, a sixth identification informationobtaining unit 973, and/or a sixth reference index assignment changingunit 974.

The slice type checking unit 642 checks a slice type of a current slice.If so, it is then decided whether to execute reordering of a referencepicture list 0 and/or a reference picture list 1 according to the slicetype. Details of the slice type checking unit 642 can be understood fromthe previous description of FIG. 10, which are not repeated in thefollowing description for the sake of brevity.

Each of the fifth and sixth identification information obtaining units971 and 973 obtains identification information indicating a referenceindex assigning method. And, each of the fifth and sixth reference indexassignment changing units 972 and 974 changes a reference index assignedto each reference picture of the reference picture list 0 and/or 1. Inthis case, the reference index can mean a view number of a referencepicture only. And, the identification information indicating thereference index assigning method may be flag information. For instance,if the flag information is true, an assignment of a view number ischanged. If the flag information is false, a reordering process of aview number can be terminated. If the flag information is true, each ofthe fifth and sixth reference index assignment changing units 972 and974 can obtain a view number difference according to the flaginformation. In this case, the view number difference means a differencebetween a view number of a current picture and a view number of apredicted picture. And, the view number of the predicted picture maymean a view number of a reference picture assigned right before. It isthen able to change view number assignment using the view numberdifference. In this case, the view number difference can beadded/subtracted to/from the view number of the predicted pictureaccording to the identification information.

Thus, to manage the reference picture list for the inter-view predictionas a separate reference picture list, a syntax structure may be defined.One embodiment of the syntax is explained with reference to FIG. 17,FIG. 18, and FIG. 19 as follows.

FIG. 17 and FIG. 18 are diagrams of syntax for reference picture listreordering according to one embodiment of the present invention.

Referring to FIG. 17, an operation of the reference picture listreordering unit for the temporal prediction 910 shown in FIG. 15 isrepresented as syntax. Compared to the blocks shown in FIG. 15, theslice type checking unit 642 corresponds to S1 and S6 in FIG. 17 and thefourth identification information obtaining unit 940 corresponds to S7.The internal blocks of the third reference index assignment changingunit 930 correspond to S3, S4, and S5, respectively in FIG. 17. And, theinternal blocks of the fourth reference index assignment changing unit950 correspond to S8, S9, and S10, respectively in FIG. 17.

Referring to FIG. 18, operations of the NAL type checking unit 960 andthe inter-view reference picture list reordering unit 970 arerepresented as syntax. Compared to the respective blocks shown in FIG.15 and FIG. 16, the NAL type checking unit 960 corresponds to S11 inFIG. 18, the slice type checking unit 642 corresponds to S13 and S16 inFIG. 18, the fifth identification information obtaining unit 971corresponds to S14 in FIG. 18, and the sixth identification informationobtaining unit 973 corresponds to S17 in FIG. 18. The fifth referenceindex assignment changing unit 972 corresponds to S15 in FIG. 18 and thesixth reference index assignment changing unit 974 corresponds to S18 inFIG. 18.

FIG. 19 is a diagram of syntax for reference picture list reorderingaccording to another embodiment of the present invention.

Referring to FIG. 19, operations of the NAL type checking unit 960 andthe inter-view reference picture list reordering unit 970 arerepresented as syntax. Compared to the respective blocks shown in FIG.15 and FIG. 16, the NAL type checking unit 960 corresponds to S21 inFIG. 19, the slice type checking unit 642 corresponds to S22 and S25 inFIG. 19, the fifth identification information obtaining unit 971corresponds to S23 in FIG. 19, and the sixth identification informationobtaining unit 973 corresponds to S26 in FIG. 19. The fifth referenceindex assignment changing unit 972 corresponds to S24 in FIG. 19 and thesixth reference index assignment changing unit 974 corresponds to S27 inFIG. 19.

As mentioned in the foregoing description, the reference picture listfor the inter-view prediction can be used by the inter-prediction unit700 and is usable for performing illumination compensation as well. Theillumination compensation is applicable in the course of performingmotion estimation/motion compensation. In case that a current pictureuses a reference picture in a different view, it is able to perform theillumination compensation more efficiently using the reference picturelist for the inter-view prediction. The illumination compensationsaccording to embodiments of the present invention are explained asfollows.

FIG. 20 is a diagram for a process for obtaining a illuminationdifference value of a current block according to one embodiment of thepresent invention.

Illumination compensation means a process for decoding an adaptivelymotion compensated video signal according to illumination change. And,it is applicable to a predictive structure of a video signal, forexample, inter-view prediction, intra-view prediction, and/or the like.

Illumination compensation means a process for decoding a video signalusing an illumination difference residual and an illumination differenceprediction value corresponding to a block to be decoded. In this case,the illumination difference prediction value can be obtained from aneighboring block of a current block. A process for obtaining anillumination difference prediction value from the neighboring block canbe decided using reference information for the neighbor block, and asequence and direction can be taken into consideration in the course ofsearching neighbor blocks. The neighboring block means an alreadydecoded block and also means a block decoded by considering redundancywithin the same picture for a view or time or a sequence decoded byconsidering redundancy within different pictures.

In comparing similarities between a current block and a candidatereference block, an illumination difference between the two blocksshould be taken into consideration. In order to compensate for theillumination difference, a new motion estimation/compensation isexecuted. A sum of absolute difference (SAD) can be found using Formulas1 and 2.

$\begin{matrix}{\mspace{79mu}{{M_{curr} = {\frac{1}{SXT}{\sum\limits_{i - m}^{m + S - 1}{\sum\limits_{j - n}^{n + T - 1}{f\left( {i,j} \right)}}}}}\mspace{79mu}{{M_{ref}\left( {p,q} \right)} = {\frac{1}{SXT}{\sum\limits_{i - p}^{p + S - 1}{\sum\limits_{j - q}^{q + T - 1}{r\left( {i,j} \right)}}}}}}} & (1) \\{{{NewSAD}\left( {x,y} \right)} = {\sum\limits_{i - m}^{m + S - 1}{\sum\limits_{j - n}^{n + T - 1}{{\left\{ {{f\left( {i,j} \right)} - M_{cur}} \right\} - \left\{ {{r\left( {{i + x},{j + y}} \right)} - {M_{ref}\left( {{m + x},{n + y}} \right)}} \right\}}}}}} & (2)\end{matrix}$where ‘Mcurr’ indicates an average pixel value of a current block; and‘Mref’ indicates an average pixel value of a reference block; ‘f(i,j)’indicates a pixel value of a current block at coordinates (i,j); and‘r(i+x, j+y)’ indicates a pixel value of a reference block atcoordinates (i+x,j+y); (m,n) indicates a current position of the block;(p,q) indicates a position of a reference block; S is the horizontalsize of the block; and T is the vertical size of the block. Byperforming motion estimation based on the new SAD according to theFormula 2, an average pixel difference value may be obtained between thecurrent block and the reference block. And, the obtained average pixeldifference value of Mcur−Mref may be called an illumination differencevalue (IC_offset).

In case of performing motion estimation to which illuminationcompensation is applied, an illumination difference value and a motionvector are generated. And, the illumination compensation is executedaccording to Formula 3 below using the illumination difference value andthe motion vector.

$\begin{matrix}\begin{matrix}{{{NewR}\left( {i,j} \right)} = {\left\{ {{f\left( {i,j} \right)} - M_{cur}} \right\} -}} \\{\left\{ {{r\left( {{i + x^{\prime}},{j + y^{\prime}}} \right)} - {M_{ref}\left( {{m + x^{\prime}},{n + y^{\prime}}} \right)}} \right\}} \\{= {\left\{ {{f\left( {i,j} \right)} - {r\left( {{i + x^{\prime}},{j + y^{\prime}}} \right)}} \right\} -}} \\{\left\{ {M_{cur} - {M_{ref}\left( {{m + x^{\prime}},{n + y^{\prime}}} \right)}} \right\}} \\\left. {= {\left\{ {{f\left( {i,j} \right)} - {r\left( {{i + x^{\prime}},{j + y^{\prime}}} \right)}} \right\} - {IC\_ offset}}} \right\}\end{matrix} & (3)\end{matrix}$where, NewR(i,j) indicates an illumination-compensated error value(residual) and (x′, y′) indicates a motion vector.

An illumination difference value (Mcurr−Mref) should be transferred tothe decoding unit. The decoding unit carries out the illuminationcompensation in the following manner.

$\begin{matrix}\begin{matrix}{{f^{\prime}\left( {i,j} \right)} = {\left\{ {{{NewR}^{''}\left( {x^{\prime},y^{\prime},i,j} \right)} - {r\left( {{i + x^{\prime}},{j + y^{\prime}}} \right)}} \right\} +}} \\{\left\{ {M_{cur} - {M_{ref}\left( {{m + x^{\prime}},{n + y^{\prime}}} \right)}} \right\}} \\\left. {= {\left\{ {{{NewR}^{''}\left( {x^{\prime},y^{\prime},i,j} \right)} + {r\left( {{i + x^{\prime}},{j + y^{\prime}}} \right)}} \right\} + {IC\_ offset}}} \right\}\end{matrix} & (4)\end{matrix}$where, NewR″(i,j) indicates a reconstructed illumination-compensatederror value (residual) and f′(i,j) indicates a pixel value of areconstructed current block.

In order to reconstruct a current block, an illumination differencevalue should be transferred to the decoding unit. Also, the illuminationdifference value can be predicted from information of neighboringblocks. In order to further reduce a bit number to code the illuminationdifference value, a difference value (RIC_offset) between theillumination difference value of the current block (IC_offset) and theillumination difference value of the neighboring block (predIC_offset)may only be sent. This is represented as Formula 5.RIC_offset=IC_offset−predIC_offset  (5)

FIG. 21 is a flowchart of a process for performing illuminationcompensation of a current block according to an embodiment of thepresent invention.

Referring to FIG. 21, an illumination difference value of a neighboringblock indicating an average pixel difference value between theneighboring block of a current block and a block referred to by theneighbor block is extracted from a video signal (S2110).

Subsequently, an illumination difference prediction value forillumination compensation of the current block is obtained using theillumination difference value (S2120). An illumination difference valueof the current block may be reconstructed using the obtainedillumination difference prediction value.

In obtaining the illumination difference prediction value, variousmethods may be used. For instance, before the illumination differencevalue of the current block is predicted from the illumination differencevalue of the neighboring block, it is checked whether a reference indexof the current block is equal to that of the neighboring block. Whatkind of a neighboring block or what value will be used is decidedaccording to a result of the checking. For another instance, inobtaining the illumination difference prediction value, flag information(IC_flag) indicating whether to execute an illumination compensation ofthe current block can be used. And, flag information for the currentblock can be predicted using the information of the neighboring blocksas well. For another instance, the illumination difference predictionvalue may be obtained using both the reference index checking method andthe flag information predicting method. These are explained in detailwith reference to FIGS. 22 to 24 as follows.

FIG. 22 is a block diagram of a process for obtaining an illuminationdifference prediction value of a current block using information for aneighbor block according to one embodiment of the present invention.

Referring to FIG. 22, information for a neighboring block may be used inobtaining an illumination difference prediction value of a currentblock. In the present disclosure, a block can include a macroblock or asub-macroblock. For instance, an illumination difference value of thecurrent block my be predicted using an illumination difference value ofthe neighboring block. Prior to this, it is checked whether a referenceindex of the current block is equal to that of the neighboring block.According to a result of the checking, what kind of a neighboring blockor what value will be used may be decided. In FIG. 22, ‘refIdxLX’indicates a reference index of a current block, ‘refIdxLXN’ indicates areference index of a block-N—both supplied by the encoder in the videosignal. In this case, ‘N’ is a mark of a block neighboring the currentblock and indicates A, B, or C. And, ‘PredIC_offsetN’ indicates anillumination difference value for illumination compensation of aneighbor block-N. If a block-C that is located at an upper right end ofthe current block is unusable, a block-D instead of the block-C may beused. In particular, information for the block-D is usable asinformation for the block-C. If the block-B and the block-C areunusable, a block-A may be used instead. Namely, the information for theblock-A may be used as the information for the block-B or the block-C.

For another instance, in obtaining the illumination differenceprediction value, flag information (IC_flag) indicating whether toexecute an illumination compensation of the current block may be used.Alternatively, the reference index checking method and the flaginformation predicting method may be used in obtaining the illuminationdifference prediction value. In this case, if the flag information forthe neighbor block indicates that the illumination compensation is notexecuted, i.e., if IC_flag==0, the illumination difference value‘PredIC_offsetN’ of the neighbor block is set to 0.

FIG. 23 is a flowchart of a process for performing illuminationcompensation using information for a neighbor block according to oneembodiment of the present invention.

Referring to FIG. 23, the decoding unit extracts an average pixel valueof a reference block, a reference index of a current block, a referenceindex of the reference block, and/or the like from a video signal and isthen able to obtain an illumination difference prediction value of thecurrent block using the extracted information. The decoding unit obtainsa difference value (illumination difference residual) between anillumination difference value of the current block and the illuminationdifference prediction value and then reconstructs an illuminationdifference value of the current block using the obtained illuminationdifference residual and the illumination difference prediction value. Inthis case, information for a neighbor block may be used to obtain theillumination difference prediction value of the current block. Forinstance, an illumination difference value of the current block may bepredicted using the illumination difference value of the neighbor block.Prior to this, it is checked whether a reference index of the currentblock is equal to that of the neighbor block. According to a result ofthe checking, what kind of a neighboring block or what value will beused may be decided.

In particular, an illumination difference value of a neighbor blockindicating an average pixel difference value between the neighbor blockof a current block and a block referred to by the neighbor block isextracted from a video signal (S2310).

Subsequently, it is checked whether a reference index of the currentblock is equal to a reference index of one of a plurality of neighborblocks (S2320).

As a result of the checking step (S2320), if there exists at least oneneighbor block having the same reference index as that of the currentblock, it is checked whether there exists one corresponding neighborblock or not (S2330).

As a result of the checking step S2330, if there exists only oneneighbor block having the same reference index of the current block, anillumination difference value of the neighbor block having the samereference index of the current block is assigned to an illuminationdifference prediction value of the current block (S2340). In particular,it is ‘PredIC_offset=PredIC_offsetN’.

If the neighbor block having the same reference index as that of thecurrent block fails to exist as a result of the checking step S2320 orif there exist at least two neighbor blocks having the same referenceindex as that of the current block as a result of the checking stepS2330, a median of illumination difference values (PredIC_offsetN, N=A,B, or C) of the neighbor blocks is assigned to an illuminationdifference prediction value of the current block (S650). In particular,it is ‘PredIC_offset=Median(PredIC_offsetA, PredIC_offsetB,PredIC_offsetC)’.

FIG. 24 is a flowchart of a process for performing illuminationcompensation using information for a neighbor block according to anotherembodiment of the present invention.

Referring to FIG. 24, a decoding unit has to reconstruct an illuminationdifference value of a current block to carry out illuminationcompensation. In this case, information for a neighbor block may be usedto obtain an illumination difference prediction value of the currentblock. For instance, an illumination difference value of the currentblock may be predicted using the illumination difference value of theneighbor block. Prior to this, it is checked whether a reference indexof the current block is equal to that of the neighbor block. Accordingto a result of the checking, what kind of a neighboring block or whatvalue will be used may be decided.

In particular, an illumination difference value of a neighbor blockindicating an average pixel difference value between the neighbor blockof a current block and a block referred to by the neighbor block isextracted from a video signal (S2410).

Subsequently, it is checked whether a reference index of the currentblock is equal to a reference index of one of a plurality of neighborblocks (S2420).

As a result of the checking step S2420, if there exists at least oneneighbor block having the same reference index as that of the currentblock, it is checked whether there exist only one such correspondingneighbor block or not (S2430).

As a result of the checking step S2430, if there exists only oneneighbor block having the same reference index as that of the currentblock, an illumination difference value of the neighbor block having thesame reference index as that of the current block is assigned to anillumination difference prediction value of the current block (S2440).In particular, it is ‘PredIC_offset=PredIC_offsetN’.

If the neighbor block having the same reference index as that of thecurrent block fails to exist as a result of the checking step S2420, theillumination difference prediction value of the current block is set to0 (S2460). In particular, it is ‘PredIC_offset=0’.

If there exist at least two neighbor blocks having the same referenceindex as that of the current block as a result of the checking stepS2430, the neighbor block having a reference index different from thatof the current block is set to 0 and a median of illumination differencevalues of the neighbor blocks including the value set to 0 is assignedto the illumination difference prediction value of the current block(S2450). In particular, it is ‘PredIC_offset=Median(PredIC_offsetA,PredIC_offsetB, PredIC_offsetC)’. Yet, in case that there exists theneighbor block having the reference index different from that of thecurrent block, the value ‘0’ can be included in PredIC_offsetA,PredIC_offsetB, or PredIC_offsetC.

Meanwhile, view information for identifying a view of a picture and areference picture list for inter-view prediction are applicable tosynthesizing a picture in a virtual view. In a process for synthesizinga picture in a virtual view, a picture in a different view may bereferred to. So, if the view information and the reference picture listfor the inter-view prediction are used, a picture in a virtual view maybe synthesized more efficiently. In the following description, methodsof synthesizing a picture in a virtual view according to embodiments ofthe present invention are explained.

FIG. 25 is a block diagram of a process for predicting a current pictureusing a picture in a virtual view according to one embodiment of thepresent invention.

Referring to FIG. 25, in performing inter-view prediction in multi-viewvideo coding, a current picture may be predicted using a picture in aview different from that of the current view as a reference picture.Yet, a picture in a virtual view is obtained using pictures in a viewneighboring that of a current picture, and the current picture is thenpredicted using the obtained picture in the virtual view. If so, theprediction can be more accurately performed. In this case, a viewidentifier indicating a view of a picture can be used to utilizepictures in neighbor views or pictures in a specific view. In case thatthe virtual view is generated, there must exist specific syntax forindicating whether to generate the virtual view. If the syntax indicatesthat the virtual view shall be generated, the virtual view may begenerated using the view identifier. The pictures in the virtual viewobtained by the view synthesis prediction unit 740 are usable asreference pictures. In this case, the view identifier can be assigned tothe pictures in the virtual view. In a process for performing motionvector prediction to transfer a motion vector, neighbor blocks of acurrent block can refer to the pictures obtained by the view synthesisprediction unit 740. In this case, to use the picture in the virtualview as the reference picture, a view identifier indicating a view of apicture can be utilized.

FIG. 26 is a flowchart of a process for synthesizing a picture of avirtual view in performing inter-view prediction in MVC according to anembodiment of the present invention.

Referring to FIG. 26, a picture in a virtual view is selectivelysynthesized using pictures in a view neighboring that of a currentpicture. The current picture is then predicted using the synthesizedpicture in the virtual view. If so, a more accurate prediction may beachieved. In step S2610, a specific syntax indicating whether to executea prediction of a current picture by synthesizing the picture in thevirtual view is extracted from the video signal. If it is decidedwhether to execute the prediction of the current picture, more efficientcoding is possible. The specific syntax is defined as an inter-viewsynthesis prediction identifier, which is set by the encoder andexplained below. For instance, at a slice layer the syntax‘view_synthesize_pred_flag’ may indicate whether to execute a predictionof a slice in a current picture using a virtual view. And, at amacroblock layer, the syntax ‘view_syn_pred_flag’ may indicate whetherto execute a prediction of a macroblock in a current picture using avirtual view. Based on the inter-view synthesis prediction identifier, avirtual view is obtained in step S2620. For example, if‘view_synthesize_pred_flag=1’, a current slice synthesizes a slice in avirtual view using a slice in a view neighbor to that of the currentslice. The current slice may be predicted using the synthesized slice.If ‘view_synthesize_pred_flag=0’, a slice in a virtual view is notsynthesized. Likewise, if ‘view_syn_pred_flag=1’, a current macroblocksynthesizes a macroblock in a virtual view using a macroblock in a viewneighbor to that of the current macroblock. The current macroblock maythen be predicted using the synthesized macroblock. If‘view_syn_pred_flag=0’, a macroblock in a virtual view is notsynthesized. Hence, in an embodiment of the present invention, theinter-view synthesis prediction identifier indicating whether to obtaina picture in a virtual view is extracted from a video signal. Thepicture in the virtual view may then be obtained using the inter-viewsynthesis prediction identifier.

As mentioned in the foregoing description, view information foridentifying a view of a picture and a reference picture list forinter-view prediction can be used by the inter-prediction unit 700. And,they can be used in performing weighted prediction as well. The weightedprediction is applicable to a process for performing motioncompensation. In doing so, if a current picture uses a reference picturein a different view, the weighted prediction may be performed moreefficiently using the view information and the reference picture listfor the inter-view prediction. Weighted prediction methods according toembodiments of the present invention are explained as follows.

FIG. 27 is a flowchart of a method of executing weighted predictionaccording to a slice type in video signal coding according to anembodiment of the present invention.

Referring to FIG. 27, weighted prediction is a method of scaling asample of motion compensated prediction data within a P-slice or B-slicemacroblock. A weighted prediction method includes an explicit mode forperforming weighted prediction for a current picture using weightedcoefficient information obtained from information for reference picturesand an implicit mode for performing weighted prediction for a currentpicture using weighted coefficient information obtained from informationfor a distance between the current picture and one of the referencepictures. The weighted prediction method can be differently appliedaccording to a slice type of a current macroblock. For instance, in theexplicit mode, the weighted coefficient information can be variedaccording to whether a current macroblock, on which weighted predictionis performed, is a macroblock of a P-slice or a macroblock of a B-slice.And, the weighted coefficient of the explicit mode can be decided by anencoder and can be transferred by being included in a slice header. Onthe other hand, in the implicit mode, a weighted coefficient can beobtained based on a relatively temporal position of List 0 and List 1reference pictures. For instance, if a reference picture is temporallyclose to a current picture, a large weight coefficient is applicable. Ifa reference picture is temporally distant from a current picture, asmall weight coefficient is applicable.

First of all, a slice type of a macroblock to apply weighted predictionthereto is extracted from a video signal (S2710).

Subsequently, weighted prediction can be performed on a macroblockaccording to the extracted slice type (S2720).

In this case, the slice type can include a macroblock to whichinter-view prediction is applied. Inter-view prediction means that acurrent picture is predicted using information from a picture in a viewdifferent from that of the current picture. For instance, the slice typecan include a macroblock to which temporal prediction, (prediction usinginformation from a picture in a same view as that of a current picture)is applied; a macroblock to which the inter-view prediction is applied;and a macroblock to which both of temporal prediction and inter-viewprediction are applied. And, the slice type can include a macroblock towhich temporal prediction is applied only, a macroblock to whichinter-view prediction is applied only, or a macroblock to which bothtemporal prediction and inter-view prediction are applied. Moreover, theslice type can include two of the macroblock types or all threemacroblock types. This will be explained in detail with reference toFIG. 28 later. Thus, in case that a slice type including an inter-viewprediction applied macroblock is extracted from a video signal, weightedprediction is performed using information for a picture in a viewdifferent from that of a current picture. In doing so, a view identifierfor identifying a view of a picture can be utilized to use informationfor a picture in a different view.

FIG. 28 is a diagram of macroblock types allowable in a slice type invideo signal coding according to one embodiment of the presentinvention.

Referring to FIG. 28, if a P-slice type by inter-view prediction isdefined as VP (View_P), an intra-macroblock I, a macroblock P predictedfrom one picture in a current view, or a macroblock VP predicted fromone picture in a different view is allowable for the VP-slice type byinter-view prediction (2810).

A B-slice type by inter-view prediction is defined as VB (View_B), and amacroblock P or B predicted from at least one picture in a current viewor a macroblock VP or VB predicted from at least one picture in adifferent view is allowable (2820).

A slice type, on which prediction is performed using temporalprediction, inter-view prediction, or both temporal prediction andinter-view prediction, is defined as ‘Mixed’. An intra-macroblock I, amacroblock P or B predicted from at least one picture in a current view,a macroblock VP or VB predicted from at least one picture in a differentview, or a macroblock ‘Mixed’ predicted using both a picture in thecurrent view and a picture in the different view is allowable for themixed slice type (2830). In this case, in order to use the picture inthe different view, a view identifier for identifying a view of apicture may be used.

FIG. 29 and FIG. 30 are diagrams of syntax for executing weightedprediction according to a newly defined slice type according to oneembodiment of the present invention.

As mentioned in the foregoing description of FIG. 28, if the slice typeis decided as VP, VB, or Mixed, the syntax for performing theconventional weighted prediction (e.g., H.264) can be modified into FIG.29 or FIG. 30. For instance, a part ‘if(slice_type!=VP∥slice_type!=VB)’may be added (2910), and if a slice type is a B-slice the if-statementcan be modified into ‘if(slice_type==B∥slice_type==Mixed)’ (2920).

By newly defining a VP slice type and a VB slice type, a format as shownin FIG. 30 can be newly added (2930, 2940). In this case, sinceinformation for a view is added, syntax elements include ‘view’ parts,respectively. For example, there is ‘luma_log 2_view_weight_denom,chroma_log 2_view_weight_denom’.

FIG. 31 is a flowchart of a method of executing weighted predictionusing flag information indicating whether to execute inter-view weightedprediction in video signal coding according to the present invention.

Referring to FIG. 31, in video signal coding to which the presentinvention is applied, in case of using flag information indicatingwhether weighted prediction will be executed, more efficient coding maybe enabled.

The flag information can be defined based on a slice type. For instance,there can exist flag information indicating whether weighted predictionwill be applied to a P-slice or a SP-slice or flag informationindicating whether weighted prediction will be applied to a B-slice.

In particular, the flag information can be defined as‘weighted_pred_flag’ or ‘weighted_bipred_idc’ in the video signal; andin particular, the slice header of a slice. If ‘weighted_pred_flag=0’,it indicates that weighted prediction is not applied to the P-slice andthe SP-slice. If ‘weighted_pred_flag=1’, it indicates that weightedprediction is applied to the P-slice and the SP-slice. If‘weighted_bipred_idc=0’, it indicates that default weighted predictionis applied to the B-slice. If ‘weighted_bipred_idc=1’, it indicates thatexplicit weighted prediction is applied to the B-slice. If‘weighted_bipred_idc=2’, it indicates that implicit weighted predictionis applied to the B-slice.

In multi-view video coding, flag information indicating whether weightedprediction will be executed using information for an inter-view picturecan be defined based on a slice type.

First of all, a slice type and flag information indicating whetherinter-view weighted prediction will be executed are extracted from avideo signal (S3110, S3120). In this case, the slice type can include amacroblock to which temporal prediction for performing prediction usinginformation for a picture in a same view as that of a current picture isapplied and a macroblock to which inter-view prediction for performingprediction using information for a picture in a view different from thatof a current picture is applied.

A weighted prediction mode may be decided based on the extracted slicetype and the extracted flag information (S3130).

Subsequently, weighted prediction according to the decided weightedprediction mode may be performed (S3140). In this case, the flaginformation can include flag information indicating whether weightedprediction will be executed using information for a picture in a viewdifferent from that of a current picture as well as the aforesaid‘weighted_pred_flag’ and ‘weighted_bipred_idc’. This will be explainedin detail with reference to FIG. 32 later.

Hence, in case that a slice type of a current macroblock is a slice typepermitting a macroblock to which inter-view prediction is applied, moreefficient coding is enabled rather than a case of using flag informationindicating whether weighted prediction will be executed usinginformation for a picture in a different view.

FIG. 32 is a diagram to explain a weight predicting method according toflag information indicating whether to execute weighted prediction usinginformation for a picture in a view different from that of a currentpicture according to one embodiment of the present invention.

Referring to FIG. 32, for example, flag information indicating whetherweighted prediction will be executed using information for a picture ina view different from that of a current picture can be defined as‘view_weighted_pred_flag’ or ‘view_weighted_bipred_flag’.

If ‘view_weighted_pred_flag=0’, it indicates that weighted prediction isnot applied to a VP-slice. If ‘view_weighted_pred_flag=1’, explicitweighted prediction is applied to a VP-slice. If‘view_weighted_bipred_flag=0’, it indicates that default weightedprediction is applied to a VB-slice. If ‘view_weighted_bipred_flag=1’,it indicates that explicit weighted prediction is applied to a VB-slice.If ‘view_weighted_bipred_flag=2’, it indicates that implicit defaultweighted prediction is applied to a VB-slice.

In case that implicit weighted prediction is applied to a VB-slice, aweight coefficient can be obtained from a relative distance between acurrent view and a different view. In case that implicit weightedprediction is applied to a VB-slice, weighted prediction can beperformed using a view identifier identifying a view of a picture or apicture order count (POC) rendered by considering discrimination of eachview.

The above flag information can be included in a picture parameter set(PPS). In this case, the picture parameter set (PPS) means headerinformation indicating an encoding mode of all pictures (e.g., entropyencoding mode, quantization parameter initial value by picture unit,etc.). Yet, the picture parameter set is not attached to all of thepictures. If a picture parameter set does not exist, a previous pictureparameter set existing right before may be used.

FIG. 33 is a diagram of syntax for executing weighted predictionaccording to newly defined flag information according to one embodimentof the present invention.

Referring to FIG. 33, in multi-view video coding to which the presentinvention is applied, in case that a slice type including a macroblockapplied to inter-view prediction and flag information indicating whetherweighted prediction will be executed using information for a picture ina view different from that of a current picture are defined, what kindof weighted prediction to execute may be decided according to a slicetype.

For instance, if a slice type, as shown in FIG. 33, extracted from avideo signal is a P-slice or a SP-slice, weighted prediction can beexecuted if ‘weighted_pred_flag=1’. In case that a slice type is aB-slice, weighted prediction can be executed if‘weighted_bipred_flag=1’. In case that a slice type is a VP-slice,weighted prediction can be executed if ‘view_weighted_pred_flag=1’. Incase that a slice type is a VB-slice, weighted prediction can beexecuted if ‘view_weighted_bipred_flag=1’.

FIG. 34 is a flowchart of a method of executing weighted predictionaccording to a NAL (network abstraction layer) unit according to anembodiment of the present invention.

Referring to FIG. 34, first of all, a NAL unit type (nal_unit_type) isextracted from a video signal (S910). In this case, the NAL unit typemeans an identifier indicating a type of a NAL unit. For instance, if‘nal_unit_type=5’, a NAL unit is a slice of an IDR picture. And, the IDR(instantaneous decoding refresh) picture means a head picture of a videosequence.

Subsequently, it is checked whether the extracted NAL unit type is a NALunit type for multi-view video coding (S3420).

If the NAL unit type is the NAL unit type for multi-view video coding,weighted prediction is carried out using information for a picture in aview different from that of a current picture (S3430). The NAL unit typecan be a NAL unit type applicable to both scalable video coding andmulti-view video coding or a NAL unit type for multi-view video codingonly. Thus, if the NAL unit type is for multi-view video coding, theweighted prediction should be executed using the information for thepicture in the view different from that of the current picture. A newsyntax may be defined in view of the above. This will be explained indetail with reference to FIG. 35 and FIG. 36 as follows.

FIG. 35 and FIG. 36 are diagrams of syntax for executing weightedprediction in case that a NAL unit type is for multi-view video codingaccording to one embodiment of the present invention.

First of all, if a NAL unit type is a NAL unit type for multi-view videocoding, syntax for executing conventional weighted prediction (e.g.,H.264) can be modified into the syntax shown in FIG. 35 or FIG. 36. Forinstance, a reference number 3510 indicates a syntax part for performingconventional weighted prediction and a reference number 3520 indicates asyntax part for performing weighted prediction in multi-view videocoding. So, the weighted prediction is performed by the syntax part 3520only if the NAL unit type is the NAL unit type for multi-view videocoding. In this case, since information for a view is added, each syntaxelement includes a ‘view’ portion. For instance, there is ‘luma_view_log2_weight_denom, chroma_view_log 2_weight_denom’ or the like. And, areference number 3530 in FIG. 36 indicates a syntax part for performingconventional weighted prediction and a reference number 3540 in FIG. 36indicates a syntax part for performing weighted prediction in multi-viewvideo coding. So, the weighted prediction is performed by the syntaxpart 3540 only if the NAL unit type is the NAL unit type for multi-viewvideo coding. Likewise, since information for a view is added, eachsyntax element includes a ‘view’ portion. For instance, there is‘luma_view_weight_l1_flag, chroma_view_weight_l1_flag’ or the like.Thus, if a NAL unit type for multi-view video coding is defined, moreefficient coding is enabled in a manner of performing weightedprediction using information for a picture in a view different from thatof a current picture.

FIG. 37 is a block diagram of an apparatus for decoding a video signalaccording to an embodiment of the present invention.

Referring to FIG. 37, an apparatus for decoding a video signal accordingto the present invention includes a slice type extracting unit 3710, aprediction mode extracting unit 3720 and a decoding unit 3730.

FIG. 38 is a flowchart of a method of decoding a video signal in thedecoding apparatus shown in FIG. 37 according to one embodiment of thepresent invention.

Referring to FIG. 38, a method of decoding a video signal according toone embodiment of the present invention includes a step S3810 ofextracting a slice type and a macroblock prediction mode, and a stepS3820 of decoding a current macroblock according to the slice typeand/or macroblock prediction mode.

First, a prediction scheme used by an embodiment of the presentinvention is explained to help in the understanding of the presentinvention. The prediction scheme may be classified into an intra-viewprediction (e.g., prediction between pictures in a same view) and aninter-view prediction (e.g., prediction between pictures in differentviews). And, the intra-view prediction can be the same prediction schemeas a general temporal prediction.

According to the present invention, the slice type extracting unit 3710extracts a slice type of a slice including a current macroblock (S3810).

In this case, a slice type field (slice_type) indicating a slice typefor intra-view prediction and/or a slice type field (view_slice_type)indicating a slice type for inter-view prediction may be provided aspart of the video signal syntax to provide the slice type. This will bedescribed in greater deal below with respect to FIGS. 40 and 41. And,each of the slice type (slice_type) for intra-view prediction and theslice type (view_slice_type) for inter-view prediction may indicate, forexample, an I-slice type (I_SLICE), a P-slice type (P_SLICE), or aB-slice type (B_SLICE).

For instance, if ‘slice_type’ of a specific slice is a B-slice and‘view_slice_type’ is a P-slice, a macroblock in the specific slice isdecoded by a B-slice (B_SLICE) coding scheme in an intra-view direction(i.e., a temporal direction) and/or by a P-slice (P_SLICE) coding schemein a view direction.

Meanwhile, the slice type is able to include a P-slice type (VP) forinter-view prediction, a B-slice type (VB) for inter-view prediction anda mixed slice type (Mixed) by prediction resulting from mixing bothprediction types. Namely, the mixed slice type provides for predictionusing a combination of intra-view and inter-view prediction.

In this case, a P-slice type for inter-view prediction means a case thateach macroblock or macroblock partition included in a slice is predictedfrom one picture in a current view or one picture in a different view. AB-slice type for inter-view prediction means a case that each macroblockor macroblock partition included in a slice is predicted from ‘one ortwo pictures in a current view’ or ‘one picture in a different view ortwo pictures in different views, respectively’. And, a mixed slice typefor prediction resulting from mixing both predictions means a case thateach macroblock or macroblock partition included in a slice is predictedfrom ‘one or two pictures in a current view’, ‘one picture in adifferent view or two pictures in different views, respectively’, or‘one or two pictures in a current view and one picture in a differentview or two pictures in different views, respectively’.

In other words, a referred picture and allowed macroblock type differ ineach slice type, which will be explained in detail with reference toFIG. 43 and FIG. 44 later.

And, the syntax among the aforesaid embodiments of the slice type willbe explained in detail with reference to FIG. 40 and FIG. 41 later.

The prediction mode extracting unit 3720 may extract a macroblockprediction mode indicator indicating whether the current macroblock is amacroblock by intra-view prediction, a macroblock by inter-viewprediction or a macroblock by prediction resulting from mixing bothtypes of prediction (S3820). For this, the present invention defines amacroblock prediction mode (mb_pred_mode). One embodiment of themacroblock prediction modes will be explained in detail with referenceto FIGS. 39,40 and FIG. 41 later.

The decoding unit 3730 decodes the current macroblock according to theslice type and/or the macroblock prediction mode to receive/produce thecurrent macroblock (S3820). In this case, the current macroblock can bedecoded according to the macroblock type of the current macroblockdecided from the macroblock type information. And, the macroblock typecan be decided according to the macroblock prediction mode and the slicetype.

In case that the macroblock prediction mode is a mode for intra-viewprediction, the macroblock type is decided according to a slice type forintra-view prediction and the current macroblock is then decoded byintra-view prediction according to the decided macroblock type.

In case that the macroblock prediction mode is a mode for inter-viewprediction, the macroblock type is decided according to a slice type forinter-view prediction and the current macroblock is then decoded by theinter-view prediction according to the decided macroblock type.

In case that the macroblock prediction mode is a mode for predictionresulting from mixing both predictions, the macroblock type is decidedaccording to a slice type for intra-view prediction and a slice type forinter-view prediction, and the current macroblock is then decoded by theprediction resulting from mixing both predictions according to each ofthe decided macroblock types.

In this case, the macroblock type depends on a macroblock predictionmode and a slice type. In particular, a prediction scheme to be used fora macroblock type may be determined from a macroblock prediction mode,and a macroblock type is then decided from macroblock type informationby a slice type according to the prediction scheme. Namely, one of orboth of the extracted slice_type and view_slice_type are selected basedon the macroblock prediction mode.

For instance, if a macroblock prediction mode is a mode for inter-viewprediction, a macroblock type may be decided from a macroblock table ofslice types (I, P, B) corresponding to a slice type (view_slice_type)for inter-view prediction. The relation between a macroblock predictionmode and a macroblock type will be explained in detail with reference toFIGS. 39,40 and FIG. 41 later.

FIG. 39 is a diagram of a macroblock prediction modes according toexample embodiments of the present invention.

In FIG. 39( a), a table corresponding to one embodiment of macroblockprediction modes (mb_pred_mode) according to the present invention isshown.

In case that intra-view prediction, i.e., temporal prediction is usedfor a macroblock only, ‘0’ is assigned to a value of the ‘mb_pred_mode’.In case that inter-view prediction is used for a macroblock only, ‘1’ isassigned to a value of the ‘mb_pred_mode’. In case that both temporaland inter-view prediction is used for a macroblock, ‘2’ is assigned to avalue of the ‘mb_pred_mode’.

In this case, if a value of the ‘mb_pred_mode’ is ‘1’, i.e., if the‘mb_pred_mode’ indicates the inter-view prediction, view direction List0(ViewList0) or view direction List1 (ViewList1) is defined as areference picture list for the inter-view prediction.

In FIG. 39( b), the relation between a macroblock prediction mode and amacroblock type according to another embodiment is shown.

If a value of ‘mb_pred_mode’ is ‘0’, temporal prediction is used only.And, a macroblock type is decided according to a slice type (slice_type)for intra-view prediction.

If a value of ‘mb_pred_mode’ is ‘1’, inter-view prediction is used only.And, a macroblock type is decided according to a slice type(view_slice_type) for inter-view prediction.

If a value of ‘mb_pred_mode’ is ‘2’, mixed prediction of both temporaland intra-view prediction is used. And, two macroblock types are decidedaccording to a slice type (slice_type) for intra-view prediction and aslice type (view_slice_type) for inter-view prediction.

Based on the macroblock prediction mode, the macroblock type is givenbased on the slice type as shown in tables 1-3 below.

In other words, in this embodiment, a prediction scheme used for amacroblock and a slice type referred to are decided by a macroblockprediction mode. And, a macroblock type is decided according to theslice type.

FIG. 40 and FIG. 41 are diagrams of example embodiments of the syntax ofa portion of the video signal received by the apparatus for decoding thevideo signal. As shown, the syntax has slice type and macroblockprediction mode information according to an embodiment of the presentinvention.

In FIG. 40, an example syntax is shown. In the syntax, the field‘slice_type’ and the field ‘view_slice_type’ provide slice types and thefield ‘mb_pred_mode’ provides a macroblock prediction mode.

According to the present invention, the ‘slice_type’ field provides aslice type for intra-view prediction and the ‘view_slice_type’ fieldprovides a slice type for inter-view prediction. Each slice type canbecome I-slice type, P-slice type or B-slice type. If a value of the‘mb_pred_mode’ is ‘0’ or ‘1’, one macroblock type is decided. Yet, incase that a value of the ‘mb_pred_mode’ is ‘2’, it can be seen thatanother macroblock type (or two types) is further decided. In otherwords, the syntax shown in (a) of FIG. 40 indicates that‘view_slice_type’ is added to further apply the conventional slice types(I, P, B) to multi-view video coding.

In FIG. 41, another example syntax is shown. In the syntax, a‘slice_type’ field is employed to provide a slice type and a‘mb_pred_mode’ field is employed to provide a macroblock predictionmode.

According to the present invention, the ‘slice_type’ field may include,among others, a slice type (VP) for inter-view prediction, a slicetype-B (VB) for inter-view prediction and a mixed slice type (Mixed) forprediction resulting from mixing both intra-view and inter-viewpredictions.

If a value in the ‘mb_pred_mode’ field is ‘0’ or ‘1’, one macroblocktype is decided. Yet, in case that a value of the ‘mb_pred_mode’ fieldis ‘2’, it can be seen that an additional (i.e., total of two)macroblock type is decided. In this embodiment, the slice typeinformation exists in a slice header, which will be explained in detailwith respect to FIG. 42. In other words, the syntax shown in FIG. 41indicates that VP, VB and Mixed slice types are added to theconventional slice type (slice_type).

FIG. 42 provides diagrams of examples for applying the slice types shownin FIG. 41.

The diagram in FIG. 42( a) shows that a P-slice type (VP) for inter-viewprediction, a B-slice type (VB) for inter-view prediction and a mixedslice type (Mixed) for prediction resulting from mixing both predictionsmay exist as the slice type, in addition to other slice types, in aslice header. In particular, the slice types VP, VB and Mixed accordingto an example embodiment are added to the slice types that may exist ina general slice header.

The diagram in FIG. 42( b) shows that a P-slice type (VP) for inter-viewprediction, a B-slice type (VB) for inter-view prediction and a mixedslice type (Mixed) for prediction resulting from mixing both predictionsmay exist as the slice type in a slice header for multi-view videocoding (MVC). In particular, the slice types according to an exampleembodiment are defined in a slice header for multi-view video coding.

The diagram in FIG. 42( c) shows that a slice type (VP) for inter-viewprediction, a B-slice type (VB) for inter-view prediction and a mixedslice type (Mixed) for prediction resulting from mixing both predictionsmay exist as the slice type, in addition to existing slice type forscalable video coding, in a slice header for scalable video coding(SVC). In particular, the slice types VP, VB and Mixed according to anexample embodiment are added to slice types that may exist in a sliceheader of the scalable video coding (SVC) standard.

FIG. 43 shows diagrams of various slice type examples included in theslice type shown in FIG. 41.

In FIG. 43( a), a case that a slice type is predicted from one picturein a different view is shown. So, a slice type becomes a slice type (VP)for inter-view prediction.

In FIG. 43( b), a case that a slice type is predicted from two picturesin different views, respectively is shown. So, a slice type becomes aB-slice type (VB) for inter-view prediction.

In FIGS. 43( c) and 43(f), a case that a slice type is predicted fromone or two pictures in a current view and one picture in a differentview is shown. So, a slice type becomes a mixed slice type (Mixed) forprediction resulting from mixing both predictions. Also, in FIGS. 43( d)and 43(e), a case that a slice type is predicted from one or twopictures in a current view and two pictures in different views is shown.So, a slice type also becomes a mixed slice type (Mixed).

FIG. 44 is a diagram of a macroblock allowed for the slice types shownin FIG. 41.

Referring to FIG. 44, an intra macroblock (I), a macroblock (P)predicted from one picture in a current view or a macroblock (VP)predicted from one picture in a different view is allowed for a P-slicetype (VP) by inter-view prediction.

An intra macroblock (I), a macroblock (P or B) predicted from one or twopictures in a current view or a macroblock VP or VB predicted from onepicture in a different view or two pictures in different views,respectively, are allowed for a B-slice type (VB) by inter-viewprediction.

And, an intra macroblock (I); a macroblock (P or B) predicted from oneor two pictures in a current view; a macroblock (VP or VB) predictedfrom one picture in a different view or two pictures in different views,respectively, or a macroblock (Mixed) predicted from one or two picturesin a current view, one picture in a different view or two pictures indifferent views, respectively, are allowed for a mixed slice type(Mixed).

FIGS. 45-47 are diagrams of a macroblock type of a macroblock existingin a mixed slice type (Mixed) according to embodiments of the presentinvention.

In FIGS. 45( a) and 45(b), configuration schemes for a macroblock type(mb_type) and sub-macroblock type (sub_mb_type) of a macroblock existingin a mixed slice are shown, respectively.

In FIGS. 46 and 47, binary representation of predictive direction(s) ofa macroblock existing in a mixed slice and actual predictivedirection(s) of the mixed slice are shown, respectively.

According to an embodiment of the present invention, a macroblock type(mb_type) is prepared by considering both a size (Partition_Size) of amacroblock partition and a predictive direction (Direction) of amacroblock partition.

And, a sub-macroblock type (sub_mb_type) is prepared by considering botha size (Sub_Partition_Size) of a sub-macroblock partition and apredictive direction (Sub_Direction) of each sub-macroblock partition.

Referring to FIG. 45( a), ‘Direction0’ and ‘Direction1’ indicate apredictive direction of a first macroblock partition and a predictivedirection of a second macroblock partition, respectively. In particular,in case of a 8×16 macroblock, ‘Direction0’ indicates a predictivedirection for a left 8×16 macroblock partition and ‘Direction1’indicates a predictive direction for a right 8×16 macroblock partition.A configuration principle of macroblock type (mb_type) is explained indetail as follows. First, the first two bits indicate a partition size(Partition_Size) of a corresponding macroblock and a value of 0˜3 isavailable for the first two bits. And, four bits following the first twobits indicate a predictive direction (Direction) in case that amacroblock is divided into partitions.

For instance, in case of a 16×16 macroblock, four bits indicating apredictive direction of the macroblock are attached to a rear of thefirst two bits. In case of a 16×8 macroblock, four bits following thefirst two bits indicate a predictive direction (Direction0) of a firstpartition and another four bits are attached to the former four bits toindicate a predictive direction (Direction1) of a second partition.Likewise, in case of a 8×16 macroblock, eight bits are attached to arear of the first two bits. In this case, the first four bits of theeight bits attached to the first two bits indicate a predictivedirection of a first partition and a next four bits indicate apredictive direction of a second partition.

Referring to FIG. 45( b), a predictive direction (Sub_Direction) of asub-macroblock is used in a same manner as a predictive direction(Direction) of the macroblock partition shown in FIG. 45( a). Aconfiguration principle of sub-macroblock type (sub_mb_type) isexplained in detail as follows.

First, the first two bits indicate a partition size (Partition_Size) ofa corresponding macroblock and the second two bits, next to the formertwo bits, indicate a partition size (Sub_Partition_Size) of asub-macroblock of the corresponding macroblock. A value of 0˜3 isavailable for each of the first and second two bits. Subsequently, fourbits attached next to the second two bits indicate a predictivedirection (Sub_Direction) in case that a macroblock is divided intosub-macroblock partitions. For instance, if a size (Partition_Size) of apartition of a macroblock is 8×8 and if a size (Sub_Partition_Size) of apartition of a sub-macroblock is 4×8, the first two bits have a value of3, the second two bits have a value of 2, the first four bits next tothe second two bits indicate a predictive direction for a left 4×8 blockof two 4×8 blocks, and the second four bits next to the first four bitsindicate a predictive direction for a right 4×8 block.

Referring to FIG. 46, a predictive direction of a macroblock isconstructed with four bits. And, it can be seen that each binaryrepresentation becomes ‘1’ according to a case of referring to a pictureat the left (L), top (T), right (R) or bottom (B) position of a currentpicture.

Referring to FIG. 47, for example, in case that a predictive directionis top (T), a picture located at a top in a view direction of a currentpicture is referred to. In case that a predictive direction correspondsto all directions (LTRB), it can be seen that pictures in all directions(LTRB) of a current picture are referred to.

FIG. 48 is a block diagram of an apparatus for encoding a video signalaccording to an embodiment of the present invention.

Referring to FIG. 48, an apparatus for encoding a video signal accordingto an embodiment of the present invention. The apparatus includes amacroblock type deciding unit 4810, a macroblock generating unit 4820and an encoding unit 4830.

FIG. 49 is a flowchart of a method of encoding a video signal in theencoding apparatus shown in FIG. 48 according to an embodiment of thepresent invention.

Referring to FIG. 49, a method of encoding a video signal according toan embodiment of the present invention includes a step S4910 of decidinga first macroblock type for intra-view prediction and a secondmacroblock type for inter-view prediction, a step S4920 of generating afirst macroblock having the first macroblock type and a secondmacroblock having the second macroblock type, a step S4930 of generatinga third macroblock using the first and second macroblocks, and a stepS4940 of encoding a macroblock type of a current macroblock and amacroblock prediction mode.

According to the present invention, the macroblock type deciding unit4810 decides a first macroblock type for intra-view prediction and asecond macroblock type for inter-view prediction (S4910) as described indetail above.

Subsequently, the macroblock generating unit 4820 generates a firstmacroblock having the first macroblock type and a second macroblockhaving the second macroblock type (S4920) using well-known predictiontechniques, and then generates a third macroblock using the first andsecond macroblocks (S4930). In this case, the third macroblock isgenerated according to a mean value between the first and secondmacroblocks.

Finally, the encoding unit 4830 encodes a macroblock type (mb_type) of acurrent macroblock and a macroblock prediction mode (mb_pred_mode) ofthe current macroblock by comparing encoding efficiencies of the firstto third macroblocks (S4940).

In this case, there are various methods to measure the encodingefficiencies. In particular, a method using RD (rate-distortion) cost isused in this embodiment of the present invention. As is well-known, inthe RD cost method, a corresponding cost is calculated with twocomponents: an encoding bit number generated from encoding acorresponding block and a distortion value indicating an error from anactual sequence.

The first and second macroblock types may be decided in a manner ofselecting a macroblock type having a minimum value of theabove-explained RD cost. For instance, a macroblock type having aminimum value of the RD cost among macroblock types by intra-viewprediction is decided as the first macroblock type. And, a macroblocktype having a minimum value of the RD cost among macroblock types byinter-view prediction is decided as the second macroblock type.

In the step of encoding the macroblock type and the macroblockprediction mode, the macroblock type and prediction mode associated withthe one of the first and second macroblocks having the smaller RD costmay be selected. Subsequently, the RD cost of the third macroblock isdetermined. Finally, the macroblock type and macroblock prediction modeof the current macroblock are encoded by comparing the RD cost of theselected first or second macroblock and the RD cost of the thirdmacroblock to each other.

If the RD cost of the selected first or second macroblock is equal to orgreater than the RD cost of the third macroblock, the macroblock typebecomes a macroblock type corresponding to the selected first or secondmacroblock.

For instance, if the RD cost of the first macroblock is smaller thanthat of the second and third macroblocks, the current macroblock is setas the first macroblock type. And, the macroblock prediction mode (i.e.,intra-view) becomes a prediction scheme of a macroblock corresponding tothe RD cost.

For instance, if the RD cost of the second macroblock is smaller thanthat of the first and third macroblocks, an inter-view prediction schemeas a prediction scheme of the second macroblock becomes the macroblockprediction mode of the current macroblock.

Meanwhile, if the RD cost of the third macroblock is smaller than the RDcosts of the first and second macroblocks, macroblock types correspondto both the first and second macroblock types. In particular, intra-viewprediction and inter-view prediction macroblock types become macroblocktypes of the current macroblock. And, the macroblock prediction modebecomes a mixed prediction scheme resulting from mixing intra-view andinter-view predictions.

Accordingly, the present invention provides at least the followingeffect or advantage.

The present invention is able to exclude the redundancy informationbetween views due to various prediction schemes between views and suchinformation as slice types, macroblock types and macroblock predictionmodes; thereby enhancing performance of encoding/decoding efficiency.

Furthermore, it will be appreciated that unless specified to thecontrary, the syntaxes, flags, etc. discussed above are set by theencoder and included in the video signal sent to the decoder embodimentsof the present invention.

While the present invention has been described and illustrated hereinwith reference to example embodiments thereof, it will be apparent tothose skilled in the art that various modifications and variations canbe made therein without departing from the spirit and scope of theinvention.

What is claimed is:
 1. A method for decoding a multi-view video data ina multi-view video data stream, comprising: receiving the multi-viewvideo data stream, the multi-view video data stream including a randomaccess picture, the random access picture including a random accessslice, the random access slice referencing only slices corresponding toa same time and a different view of the random access picture; obtainingidentification information representing the multi-view video data streamincluding initialization information of a reference picture list for therandom access slice; obtaining the initialization information of thereference picture list for the random access slice based on theidentification information, the initialization information representinga reference relation between a plurality of views, the initializationinformation including view number information and view identificationinformation; initializing the reference picture list using the viewnumber information and the view identification information; obtaining adifference value from the multi-view video data stream according to thetype information, the difference value representing a difference betweeninter-view reference indices in the initialized reference picture list;determining an assignment modification value for modifying an inter-viewreference index assigned to a reference picture in the initializedreference picture list according to the difference value; modifying theinitialized reference picture list for inter-view prediction using thedetermined assignment modification value; determining a prediction valueof a macroblock in the random access picture based on the modifiedreference picture list; and decoding the macroblock using the predictionvalue, wherein the initialization information is obtained from anextension area of a sequence header.
 2. The method of claim 1, whereinthe view number information indicates a number of reference views of therandom access picture, and the view identification information providesa view identifier of each reference view for the random access picture.3. The method of claim 1, wherein the multi-view video data includesvideo data of a base view independent of other views, the base viewbeing a view decoded without using inter-view prediction.
 4. The methodof claim 1, wherein the identification information is obtained based onNAL (Network Abstract Layer) unit type, the NAL unit type indicatingsequence information of the multi-view video data.
 5. The method ofclaim 1, wherein the difference value is obtained from a slice header.6. The method of claim 1, wherein the determined assignment modificationvalue is used to assign an inter-view reference index to the randomaccess picture in the initialized reference picture list.
 7. The methodof claim 1, wherein the assignment modification value represents avariable associated with a view identifier of the reference picture inthe initialized reference picture list.
 8. The method of claim 1,wherein the position of any other remaining pictures are shifted tolater in the initialized reference picture list.
 9. An apparatus fordecoding a multi-view video data in a multi-view video data stream,comprising: a nal parsing unit receiving the multi-view video datastream, the multi-view video data stream including a random accesspicture, the random access picture including a random access slice, therandom access slice referencing only slices corresponding to a same timeand a different view of the random access picture, and obtainingidentification information representing the multi-view video data streamincluding initialization information of a reference picture list for therandom access slice, and obtaining the initialization information of thereference picture list for the random access slice based on theidentification information, the initialization information representinga reference relation between a plurality of views, the initializationinformation including view number information and view identificationinformation; a decoded picture buffer unit initializing the referencepicture list using the view number information and the viewidentification information, and obtaining a difference value from themulti-view video data stream according to the type information, thedifference value representing a difference between inter-view referenceindices in the initialized reference picture list, and determining anassignment modification value for modifying an inter-view referenceindex assigned to a reference picture in the initialized referencepicture list according to the difference value, and modifying theinitialized reference picture list for inter-view prediction using thedetermined assignment modification value; an inter-prediction unitdetermining a prediction value of a macroblock in the random accesspicture based on the modified reference picture list, and decoding themacroblock using the prediction value, wherein the initializationinformation is obtained from an extension area of a sequence header. 10.The apparatus of claim 9, wherein the view number information indicatesa number of reference views of the random access picture, and the viewidentification information provides a view identifier of each referenceview for the random access picture.
 11. The apparatus of claim 9,wherein the multi-view video data includes video data of a base viewindependent of other views, the base view being a view decoded withoutusing inter-view prediction.
 12. The apparatus of claim 9, wherein theidentification information is obtained based on NAL (Network AbstractLayer) unit type, the NAL unit type indicating sequence information ofthe multi-view video data.
 13. The apparatus of claim 9, wherein thedetermined assignment modification value is used to assign an inter-viewreference index to the random access picture in the initializedreference picture list.
 14. The apparatus of claim 9, wherein theassignment modification value represents a variable associated with aview identifier of the reference picture in the initialized referencepicture list.
 15. The apparatus of claim 9, wherein the position of anyother remaining pictures are shifted to later in the initializedreference picture list.